RFC: Expose User Requested Image Metadata

[wpferguson]

The Problem

• lines of code - adding a new metadata field to exif.cc is 100 - 200 lines of code added to a file that is already 5000 lines of code. add highlight preservation modes for Canon/Olympus/Pentax #19375, Add support for extracting, storing, and filtering on Fujifilm Film Modes #19622
• must be supported by exiv2 - Canon added Roll and Pitch data to the metadata several months ago. I extract and use it to auto straighten my images, but I have to use exiftool because exiv2 doesn't support it.
• time to release (usability) - If I open an issue today (5.4. release date) the earliest I will see a released solution is 6 months. At the time a user opens an issue to get a new metadata field exposed is when they need it.
• store in image table? - Should the metadata be stored in the image table. I struggled with this when I read Add support for extracting, storing, and filtering on Fujifilm Film Modes #19622. I looked at the information that was stored in the image table and realized that information is required by the modules to process the image. Storing a field such as Fujifilm Film Modes or Canon Picture Modes, which are how the included JPG is processed, in the image table serves no purpose for darktable since darktable can't apply processing based on this value. However, the user feels that they have a use and would like to have it. We just need to store it somewhere else (RFC: Functional Tagging #19971).

The Ideal Solution

• The user wants a new piece of metadata
• They lookup the exif tag information
• They open the (nonexistent) add metadata tool and enter a label and the exif tag information
• They hit enter and the add metadata tool figures out which engine to use, adds a field to the image information module and figures out where to store it so the user can access it.
• The added metadata settings are saved so it's always available from now on.

The Possible Solution

When @phweyland upgraded the image information module he made it possible for Lua to add fields and register a callback to update them. I wrote a proof of concept script to test it and got it working.

I realize a script might not work for all because some users have problems installing the scripts. However, #19899 should fix that. I've already got it working so I'm confident it will be ready for 5.6. With that the solution becomes "built-in", with the exception of exiv2 and exiftool. The exiv2 library is included in darktable, so I just need to make it accessible from Lua. With the lua-scripts built in, we can set certain scripts to auto start and thus they just become part of darktable.

So with all of the above....

• the user opens the add metadata module
• they specify the exif tag and label and possibly engine to use.
• add metadata adds a field to the image information display and the user can choose to make it visible or not
• when an image is hovered the callback checks to see if the metadata value is cached or not. If not then it checks for a metadata tag and gets the value, then caches it. If no tag, then exiv2 or exiftool is invoked to get the value and it is saved as a tag and cached.
• the user can access the metadata at darktable|metadata|<label>|<value>. The user can also access the variable $(CATEGORY[0,darktable|metadata|<label>]) which will give the value

[horazont]

Thanks for writing this! In general, I can see that it's useful to keep the tables clean. Compared to #19622, I have a couple questions:

• Can we include the ability to sort and filter based on this user metadata?
• Can we somehow allow for forcing the read of that metadata for all existing images (yes, that may take long with a large catalogue) so that it's available for sorting/filtering?
• Can we allow users to merge EXIF tags into the same field? In the case of the Fujifilm FilmMode tag, I made the code use the Saturation field if and only if the FilmMode tag was absent/empty, because that's where the FilmMode is stored for monochrome modes. For usable filtering, this kind of coalescing would be required.

[wpferguson]

Can we include the ability to sort and filter based on this user metadata?

If we store the extra metadata in a tag, then the existing collection sorting and filtering will be usable

Can we somehow allow for forcing the read of that metadata for all existing images

Yes. We can give a shortcut to scan a selection or collection and give the option to scan the collection on import.

Can we allow users to merge EXIF tags into the same field?

Yes, but that might have to be something separate, since that's a user specific workflow. But it is possible and not difficult.

[horazont]

Thanks for the reply!

I just now clicked through to the Functional Tagging RFC and now a couple things from this RFC made more sense (the darktable|metadata|…|… syntax in particular).

Is there a reason not to use the existing metadata infrastructure which I touched in #19656 for this? It seems more appropriate on the first glance.

(Not sure if this comment isn't more appropriate on #19971 actually...)

[wpferguson]

Is there a reason not to use the existing metadata infrastructure

It's tied to the image table, so all the underlying SQL queries against that.

Though if we have a "standard" (darktable|metadata) perhaps we can work with the underlying SQL to look in 2 places

[horazont]

It's tied to the image table, so all the underlying SQL queries against that.

Just to make sure we're talking about the same thing: I meant the metadata infrastructure which uses the main.meta_data table. AIUI, the ties between main.meta_data and main.images are similar to the ties between data.tags, though data.tags has an intermediate table main.tagged_images to represent the n:m relationship.

I see some value in using main.meta_data for this purpose (won't have to worry about keys or values which contain |, for example).

[wpferguson]

The big problem for me is exiv.cc is already 6.6K lines of code and each time we add a metadata field it's another 150+ lines of code. I don't see it as sustainable much longer (I'm not even sure it is now). So, unless it's a field that darktable requires so that it can process the image I would be against adding it. At the same time, I can see the user's need to be have access to metadata so that they can make processing decisions.

So, we need to find a solution where each user can add the metadata they want for their specific workflow. We also need to find a solution to "I want my metadata now, not 6 months from now" :-)

I think I'm on the right track for a solution, a generic way to add metadata, but the implementation may need some tweaking. Right now I'm mostly focused on working with what exists to minimize the impact and to get it into the hands of the users ASAP.

The worst case would be I've figured out one way not to do it :-).

The probable case will be "it works, but...". Then we'll go through a series of tweaks to get closer to what's desired. Then once the details are ironed out maybe someone ports it to master. Or not.

won't have to worry about keys or values which contain |

Keys are under the control of the user, so that shouldn't be a problem. I've never seen a metadata value with |, though I suppose it's possible.

Now I have a question. If you want to know the film simulation information, does it matter if you access it as a string in a tag or a string in the image metadata. Either way it's tied to the image. The one difference would be the tag is stored in the XMP file, so if you imported the image/xmp pair on a different system the tag would automatically be populated.

[horazont]

The big problem for me is exiv.cc is already 6.6K lines of code and each time we add a metadata field it's another 150+ lines of code. I don't see it as sustainable much longer (I'm not even sure it is now). So, unless it's a field that darktable requires so that it can process the image I would be against adding it. At the same time, I can see the user's need to be have access to metadata so that they can make processing decisions.

I think we're still not on the same page here.

The mechanism for users to add custom metadata for Darktable to collect from EXIF is already there (it is buggy though which #19656 fixes).

No code changes are required to collect new fields.

Maybe @zisoft can explain it better than me or tell me where I'm misunderstanding you.

[wpferguson]

The mechanism for users to add custom metadata for Darktable to collect from EXIF

I don't think it collects it. I think that is a way for you to create a custom metadata field and supply the value. It's not going to pull information from the image unless exiv2 is modified to supply it.

[horazont]

Okay, fair enough, it needs a code change to pull information from exiv2, though that code change is minor (#19656). The discussion in #19622 (different number!) made me believe that the intent of the metadata editor's customization options was to support pulling arbitrary metadata from EXIF in the first place, supported by the fact that it allows to literally pick EXIF tags from a list.

In any case, code will have to be written to pull metadata from EXIF and put them into some kind of darktable database. That can be as tags darktable|metadata|<key>|<value> or using the existing metadata key/value storage.

To me, it makes a lot of sense to reuse the existing key-value based storage instead of inventing a new key-value based storage on top of a tagging mechanism. (Also, the code needed to pull data from EXIF is already there, in #19656, and it's not in any way specific to Fujifilm.)

Am I not making sense here?

[wpferguson]

What I thought the metadata editor module was for was adding things like IPTC metadata (metadata about the image, not metadata from the image).

code will have to be written to pull metadata from EXIF and put them into some kind of darktable database

Most of that code already exists, which is why I suggested tagging. The only thing that needs to be written is a script, most of which already exists from my previous times of playing with image metadata. The only possible change to master would be giving Lua access to the internal exiv2 library so we don't need to go out to the OS to get information.

it makes a lot of sense to reuse the existing key-value based storage instead of inventing a new key-value based storage

The existing key-value store only consists of information required to process the images. If we open it to storing any image metadata wanted by the user then darktable processing slows down because we're handling possibly huge records.

I'll go back and look at #19656

[wpferguson]

it makes a lot of sense to reuse the existing key-value based storage

Are you suggesting using the meta_data table, not the images table?

EDIT: The problem with using the meta_data table is that you can't use variables to expose the information unless there is some magic going on when you add a metadata field.

[horazont]

Are you suggesting using the meta_data table, not the images table?

Yes.

EDIT: The problem with using the meta_data table is that you can't use variables to expose the information unless there is some magic going on when you add a metadata field.

You mean variables in things like the export file path? The magic is already there indeed, I used that the other day to have the $(Xmp.dc.title) in my filename, and from the tooltip it looks as if it'll also work with other metadata fields.

EDIT: just tried it. Added Exif.Fujifilm.FilmMode to the metadata editor, put a value in, used it in the export filename template as $(Exif.Fujifilm.FilmMode) -> worked.

[horazont]

From my perspective, the only bits missing in the meta_data infrastructure are:

1. What exif: implement reading user-defined EXIF tags from files #19656 does: actually read the data from EXIF
2. Maybe Lua API exposition, I haven't checked that yet