 |
 |
On Mon, Apr 15, 2019 at 11:20 AM Thomas Dunbar <[log in to unmask]> wrote: > Hello everyone, > > I'm working on a proof of concept web application for common library data > conversions with support for large files. > The application is build using a serverless architecture, which allows me > do this at scale and at low cost. > Love the concept -- I tried a few conversions, including some north of 200MB. Overall, it worked impressively. Not having to download software is cool because you don't always have the ability to download software or might need to do something using a cell phone. For me personally, the chief needs driving conversions are : 1) To perform fixes in a format that's easier to work with (e.g. no one fixes in binary MARC) and convert back; 2) analysis -- i.e. identify records/elements that have or don't have X; and 3) migrations (which have required further manipulation in every single case). In other words, manipulations and partial extractions. In the context of these use cases, delimited text, plain text, XML, MARC, and JSON (to a lesser extent) dominate conversion needs. Regarding the MARC to text conversion, delimited text conversions need a subdelimiter for repeated fields as this is what often must be loaded into another system, presented in a table to someone, etc. -- the current method which adds more lines will cause trouble for anyone without coding skills. On a related note, considering the indicators part of the field makes philosophical sense but it creates practical problems (especially with nonrepeatable fields). For example, it scatters the 245 titles over as many indicator variations that exist making the simple task of generating a list of titles trickier than it should be. MARC already has a huge number of fields, so when the indicator permutations are combined with separate fielding for repeated fields, it takes no time at all to get many hundreds of fields with files that aren't that big -- something headache inducing even for those with mad skilz. One thing you'll want to think about as you develop the tool is what the people use it to accomplish. In my experience, conversions set you up for what you were really doing rather than being objectives in their own right. But again, very cool. kyle