Project Updater

Very nice!

I am trying to bring Image Tools to be compliant. I have a few thoughts to help developers who want to build and test for building a compliant package:

* It would be useful to have the equivalent of a VersionCheck function.

command: Updater#VersionCheck("Image Tools")
--> your installed version of Image Tools is 1.00
--> the current version on Igor Exchange is 2.11

* It would be useful to have the equivalent of a TestDownLoad function.

command: Updater#TestDownLoad("Image Tools")
--> the ZIP package Image Tools version 2.11 is now downloaded to Desktop:...

I have not gotten so far to see how well this would work to update my entire set of files in the Image Tools package. Because my package folder has a collection of files, I suspect however that what I will want to be able to set is the equivalent of this:

Static Constant kALLOW_AUTOTEXTFILE_UPDATES=0

Once I get the updater working, I'll have a notice at the next update of Image Tools (to version 2.20).

Good suggestions.

For now, changing the value of the static constant kChatty to 1 in updater.ipf adds a series of doAlerts that allow the user to quit at various points along the path to updating. You can check for new version and refuse to download any new package, or make the download and refuse to overwrite any files, and refuse to delete the downloaded files.

Note that if you run the check for releases  from the Misc menu you can cycle through eligible open procedures and check for releases. You should see something like this for each file:

UserProcLoader: Checking for updates... Current version (1.31) is the most recent release

The IgorExchange version info isn't written to history.

You can also make a trial download of any package using the installer GUI. If you don't provide an install location and refuse to delete the temporary install folder you should find the download on the desktop. But testing a download with the installer is not the same as testing an update, because the installer doesn't require compliance.

I can see that it would be helpful to provide some kind of compliance check, and I'll think about how to implement that.

Note that for image tools you will have to provide the download location in the file itself:

I never asked anyone at WM whether we could rely on the web site having a constant format over time. It seems that older projects can be found using the project short title (which in any case is optional), but recent releases cannot. In fact, it appears that updater is the last project that can be addressed in this way:

https://www.wavemetrics.com/node/8197

is the same as

https://www.wavemetrics.com/project/Updater

If it turns out that this project has a life beyond version 8 of Igor Pro, it would be great if IgorExchange had a more easily and quickly parsed directory of available project downloads...

Never mind. It all works perfectly now. I had mis-typed the location of the Image Tools package. It has to be /project-releases/ rather than projects/ or node/ (that I get when I look at the location of the package directly on Igor Exchange).

For those who want to test this for your own package, here are some additional insights:

* Do the modifications to your package as described. Presumably your version will be > 0.00
* Upload your package to Igor Exchange (ZIP archive).
* Set all the versioning information on the Igor Exchange site as directed.
* Go back to your local package and change its #pragma version = 0.00
* Run the Misc->Tony's Updater: Check for new releases menu option

--> You should get a notice that your package is (to be) downloaded and you should get a notice that your files will be overwritten.

This works also for a package that has a collection of installation files (e.g. Image Tools).

My only closing request now is for a VersionCheck(...) function to report back the installed and the current version. For the longer haul, are you at all familiar with the TeXLive installation utility on macOS? I've attached a panel layout as an example. I have long wished that Igor Pro packages could be managed in a comparable way.

In any case, this is a great job Tony! You've created isa superb way for developers to get their packages out with far less headache. I especially like using this coupled with the UserProcLoader package.

Thanks, I'm glad you got it to work.

I've not looked at TeXLive, but I have seen something similar for a python IDE. Because Igor allows many different formats for packages it's hard to imagine how a general package manager could work. It would be hard to guess which files are part of which package, which require XOPs or other custom files, etc. But for simple packages that have a main procedure file that has updater headers, it may be possible to walk the user procedures folder and check for updates. 

The installer does a reasonable job of installing simple packages. More complex installations are possible by means of an ITX script. But I don't think I included an option for running scripts as part of an update.

I am now thinking it would make sense to use the temporary directory provided by the OS as the download location. I'll change that for the new release, and I've replaced the 'Check for new releases' menu item with 'Check Release Versions' and 'Install New Releases'. The first prints version info to history without downloading new releases, the second checks for and downloads new releases. Is that sufficient for version checking?

> I am now thinking it would make sense to use the temporary directory provided by the OS as the download location. I'll change that for the new release,

I think that this will or will not matter primarily depending on how conversant the end user is. Some developers/folks may want to keep the downloaded package "exposed" (on the Desktop) so that their users can find them again if something goes wrong. Others may prefer to use the "hidden" OS temporary directory always.

Perhaps have a Static Constant kdownloadlocation 0/1 for the developer and a function "SetDefaultDownloadFolder(0/1) for the user with 0 - to the Desktop and 1 - to the OS temporary.

> and I've replaced the 'Check for new releases' menu item with 'Check Release Versions' and 'Install New Releases'. The first prints version info to history without downloading new releases, the second checks for and downloads new releases. Is that sufficient for version checking?

This is perfect.

Thank you Tony for your ongoing efforts to maintain and improve upon this package. I greatly appreciate its utility and ease of use.

Tony, there seems to be a small bug with the right-click menu to get the website. The popup menu does not fire, because the eventMod field checking on line 4243 is off. The check looks for 17 (bit 0 and 4), but at least here on Igor 9 / Win10 I get 16 (only bit 4). I don't know if this is different for older Igor versions or Mac, but changing this to just check bit 4 should work always: if (s.eventMod & 2^4).

I really like this project. It would be really great if this could come preinstalled to Igor. I get the feeling a large percentage of users don't even realize that this site exists, and much less that there are a lot of useful projects. :)

Thanks! Version 4.1 hopefully fixes the right-click bug.

I'm not sure that something like this should be preinstalled. User code - or, at least, the code in my projects - is potentially less robust than WM add-ons. It's important for users to know that when a user project causes problems it's not Igor that is broken. Having it built-in could lead to WM support requests every time I release something with a bug!

As a project author, you may be interested in the code at the end of the procedure that helps prepare project releases. I have the following in my 'default includes' procedure file so that I can easily access that code:

Thanks Tony, very interesting. I will keep this tool in mind the next time I do a release.

Yeah, I didn't mean that WM should just slap on your project onto official releases. I was thinking that maybe in the future, and with the help of some official supervision, this project or something similar could find its way into Igor. It would be a great addition for a better integration of community projects.

I too would like to see better integration of package management tools into Igor. Coming from the LaTeX/TeXLive paradigm, I was always a bit befuddled by how to manage what I had or did not have installed for Igor. This package from Tony is a significant step in the much-needed direction for helping me keep up with what I have.

The opposing side to us asking WM to add this package management tool is that those folks who post packages for distribution should be held to a modicum of standards for how their package is made available and provides help for the end user. By example, I need only go to TeXShop's menu to get help for a package and I am offered a respectable help file for the specific package. Would that all Igor Pro packages take the same approach (yes, even some of mine). At one point, way back when time seemed far more free-flowing in my schedule, I had started something along this line with Package Tools.

Indeed, it might be a problem if users just install a community project from within Igor and then complain to WM about problems, not knowing that they are not responsible. At least a message of caution should be displayed then. How does the TeX community handle this? I guess if the WM staff would need to at least rudimentarily check uploaded projects, this might instead backfire by slowing down releases and sucking up resources from the team. Maybe another category of 'officially approved' projects next to the normal projects list would make this work. This way submissions can live here, and get elevated to the 'official' category / appear in Igor once somebody had time for a check of minimal requirements. I understand that this is a tricky problem, but as you said, it would be nice to have a package manage just like your approach or now the Updater.

The TeX community has a mailing list, the TeX StackExchange. Folks also make the direct reports through CTAN (a repository of packages) or the email link for the package author (for example, I have a follow up recommendation to make to the author for the probsoln package).

The CTAN site has a set of package guidelines and an upload manager + committee to police behind.

https://ctan.org/file/help/ctan/CTAN-upload-addendum

They are volunteers (TeX is an open-source / non-commercial development). An equivalent for Igor Pro might therefore be to have a volunteer committee. Their work could be a simple as giving packages a "Good Seal of Approval" on their own free time as you note to as difficult as being the front-end gatekeepers who may reject packages that do not follow at least a minimum set of standards for code structure (e.g. for compatibility with this package).

WaveMetrics could sponsor the committee with mugs, T-shirts, and a plaque of all icons used in Igor since its inception (yes, I would like this one especially to hang on my wall). :-)

I am curious how commercial apps such as Mathematica, MatLab, or Maple police behind their user contributions.

Jeffrey, sorry for the very late reply and thank you for your explanation. I see that (probably thanks to the open nature) contributions are more organized. Yes, it might be nice to have a volunteer team for Igor to look over projects. The number of project releases seems to have died down a bit in the last years. With 4-8 projects released per year, this might be not too much of a burden, depending on the size of the projects. I wonder how many people WaveMetrics could bring together for this. Is there still merchandise like the mug available by the way? Haven't seen one in years.

 Is there still merchandise like the mug available by the way? Haven't seen one in years.

We don't sell anything physical anymore due mostly to sales/VAT tax laws. Trying to keep up with the laws on physical shipments so we could sell a few mugs a year (more or less at cost, if I recall correctly) just wasn't worth it.

If you have time to burn, you might look into using Amazon or Etsy or the like as marketplace that handles this for you.  Probably not worth the effort though.

Andy

> I wonder how many people WaveMetrics could bring together for this.

The LaTeX community is voluntary. Perhaps then ... how many people might simply be willing to volunteer on an ad hoc approach?

(me) + ... ???

I'm definitly interested in getting more package mangement features for IP available. Regarding the suggestions here floating around. I don't think the package management team should do any code review at all. Doing code review is a very time consuming task when done properly. And if you don't do it propertly there is also no benefit. It is the users responsibilty to check that the code she installs is fine to use.

> The number of project releases seems to have died down a bit in the last years.

Most projects I'm developing have their more historic versions on igorexchange, everything current is on github. I occasionally sync them to igorexchange, but as that can not be automated, I tend to not do it very often.

Regarding existing package management solutions in other projects. I do like how brew, MacOSX, works, TexLive works also well. A nice example for existing package management systems is also conda.

I imagine something like conda's meta.yaml [1] file for providing details about the package and then some code in IP, maybe with a custom XOP, to do the installation/upgrade/removal/dependency handling. Their feedstock/smithy approach is also cool, but overkill for IP.

[1]: https://github.com/conda-forge/cpptango-feedstock/blob/master/recipe/me…

I would prefer not to have the report of updates writing into the history of an experiment. Also, I can see where someone may want to track backward on their history of changes to packages (e.g. for troubleshooting). This is not easily possible in the current approach.

Could the update report be moved to its own notebook log file that is subsequently (re)saved to the package folder?

It should be straightforward to redirect any history output into a log file.

How about this:

Create a file - something like InstallHistory.txt - in the same location as install.log. Install.log is the file that records the project info and filelist during installation. Put a button in the settings panel that opens the install history as a notebook. The file will contain a log of all installation and updating activity.

Do you think the current level of reporting is about right? I would add timestamps to the current history output and suppress the output from unzip/unziparchive which, on the mac at least, is a bit long. Maybe add an option for mirroring output to the experiment history.

The approach you suggest is reasonable. Timestamps are good things. No need to report the unzip information. A checkbox or similar to allow users to select " [ ] report install to history" would offer the equivalent of backward compatibility.

All of this sounds good!

It's all in version 4.21

Just found this - fantastic! Thanks a million... I'll try to add my project (once it is published) here...

Tony, neat trick with the install script. I might grab this idea when distributing my projects to people. :)

@chozo

There is an undocumented helper function included in updater for generating install scripts for your projects:

updater#CreateInstallScript()

The install scripts generated in this way contain a header line

X // standalone install script

which identifies them as install scripts so that when the GUI is used to install or update a project the script is discarded.

Tony, I see. Thanks for the info. I will try it.

Tony -- Can you offer a suggestion on this idea?

Suppose that I want users to install a specific selection of packages from Igor Exchange. I can certainly provide step-by-step instructions, starting from 1) install the Updater package (in this way), 2) go to this menu, 3) go to this tab, 4) choose this package and install, 5) choose this package and install, ...

What I imagine doing instead is creating an Igor Pro experiment. When the experiment is run, it will a) validate that required packages are installed and b) validate that installed packages are up to date.

Can this approach done by hooking into the Updater package within the installation experiment? If so, how?

Specifically, I might imagine an installation experiment that has the code below in its main procedure file.

Yes, that certainly is possible. There are actually a few different options.

You could try updater#install(packageList), where packagelist is a string list of project ID numbers. It used to be that you could use the short title (e.g., install("Spidergram")), but for newer projects it has to be the more cryptic project ID. This is probably the most simple way, though it may take some time to grab all the files from the web. I have to admit that I haven't used this in a long time, I hope it still works! It looks like the Install() function should return the number of successful installs.

Another option is updater#InstallFile(filePathList), where filePathList is a list of local files. This was designed for distributing packages, and can be run from an itx that you include with the projects to be installed. Or it could be run from an experiment, as you envisage.

A third option is to move all the files from a local folder to User Procedures using a script - I distribute my own package of projects like that. That way you don't need updater, and can even include updater with the other files. The projects won't be added to the install log until the next time they are updated from IgorExchange. If you're interested in this option I can send you an example.

If I understand, the third option requires me to post a folder (ZIP archive) with updated procedure files each time they are updated. I would subsequently need to send a notice to the users to inform them that they can grab the update folder and run the update installer.

I will consider this option, especially since some of the procedures that I want to distribute may not be ones that I want posted at IgorExchange.

I should also test Updater#Install(...) option. It speaks in part to a next step ... a companion Updater#Validate(...) function. The Validate function would validate the input list of required IgorExchange packages against the current local install base, update any outdated packages as needed, and then, also as needed, install any missing packages using the Updater#Install(...) function.

I may be in touch directly via email likely in the next week or so.

Thanks!

jjweimer wrote:

If I understand, the third option requires me to post a folder (ZIP archive) with updated procedure files each time they are updated.

No, the third option is just a way to distribute a selection of procedures that need to be saved in the user's Igor Pro User Files folder. If as part of that process the updater procedure is saved in the Igor Procedures folder, and provided the user uses the default updater settings, any IgorExchange-hosted updates will be offered as an update to the user - that's the normal function of this project (I am assuming - or rather I know - that your procedures have updater headers).

>  any IgorExchange-hosted updates will be offered as an update to the user ...

I'll rephrase. Any package procedure files that are not posted on IgorExchange will still have to be sent separately. But the procedures that are not on IgorExchange will be in the local folder, and your install script will move them in place.

Sounds like an excellent option.

When you have a moment, I'd be glad if you could send me a ZIP archive with a demonstration -- weimerj at uah dot edu.

@jjweimer

One other thought: Updater knows about projects that were installed using the gui, and about projects that have 'updater headers' in a procedure file. If you have projects hosted on IgorExchange as pxp files it would be better to install these via updater so that they are added to the installation log and can be monitored for updates.

Yes, I will keep the updates through IgorExchange as appropriate.

Tony, I post that here, but this seems to be a general issue with your projects. You have the lines ...

in your code. However, even though these are declared as strconstant, the assignment is missing double quotes. The latest nightly (build 56629) complains about these things. Better fix this before the next patch is out.

Thanks, chozo, for the heads up. That's a bit of a killer for this project in particular! I've made new releases for five projects that contain unquoted strconstants. I fear though that many users will ignore the updates and end up with broken code that can only be fixed by hand. Ouch.

Yes, I was thinking this could be a potential roadblock for automatic updates / bugfixes. Maybe there will be an option (e.g., using SetIgorOption) to temporarily disable these new checks. You could then provide an itx script to disable the check and force an update for affected users. But then, there is maybe enough time until the next version for most users to pick up the fix.

@chozo: Thanks for mentioning the issue with unquoted strconstants. I've notified the responsible person internally of the breakage so we can reconsider whether the change is wise.

I've reverted that change to Igor 9 for the next Nightly Build.

I would welcome more stringent tests as an additional option. Or could this maybe made into a runtime check (e.g. upon first use of such constant)? This would prevent breaking compilation while still offering the benefits. I also noticed that error handing, e.g., for out-of-range checks seems to have changed. At least some botched code involving try-catch loops is less willing to cooperate now... :)

Thanks, Jim, for the reprieve.

I've unwittingly taken advantage of the liberal behaviour of Igor's compiler for a long time :)  Hopefully by the time this change is eventually introduced most code will have been patched.

Tony, for some reason the Updater tells me that there are updates even though all projects are then labeled as 'up to date'. Did something change in this regard lately? I can investigate further if you tell me where to look.

If you didn't already try this, could you open the settings panel and click 'show history' and have a look at the last few lines to see if there is a project listed there as needing an update?

Then click on 'show file', and rename the cache file (cache.txt, it should always be safe to delete this file, but we will keep a copy in case it contains a clue as to what is going on).

In the updates tab, click on the green reload button to see if updater still thinks there is something to be updated.

Sorry for the inconvenience.

Also, in the updates tab, use the popup menu to take a look at both 'installed files' and 'user procedures folder'.

The version numbers for installed files are the version taken from the web page during installation, those in the user procedures folder check are taken from the files. They should be the same unless a project has been released with out-of-sync version numbers in file and on the web.

Tony, I updated to your new version 5.0, but project updates still do not seem to work properly, or do I need to reset something (cache, history etc.)? I am working on Windows 10 with Igor 9.0.6.1 and get the error in the attachment.

Can you confirm that version 5.0 was actually installed? After the change to the new server, updater 4.x cannot find the files that it needs to update projects, including itself! The solution is to go into the settings, click on "Repair Updater", and agree to any downloads.

I tested version 5 on Mac and Windows and didn't find any problem, so if version 5 is not working for you I will have to dig a little deeper to find out what is going wrong. Sorry about that.

I can confirm that I am using version 5.0. I also did the Repair Updater thing, but it did not help. I just tried to install a new project from the list and this apparently works. Also, I deliberately downgraded another package (Baselines) to see if this would fail, but the update installed just fine. So I guess it is a problem specifically with the CalcPad project. Maybe you want to have a look in this direction then...

Then try clearing the cache (button in the settings). Perhaps version 4.x of Updater recently cached an incorrect location of the needed file, and version 5 is trying to use the wrong location rather than looking up the location taking into account the changes in the server.

So I updated to version 5.03 of the updater and cleared the cache. Now everything seems to work. Thanks for your quick response.

Great, I'm glad that it is working.

For anyone using a version of this project older than 5.0, you will need to need to update to the most recent version using the 'repair' function. Open the settings, click on "Repair Updater", and accept any options to download and install a working version.

Just FYI ... Some initial issues repairing and updating from 4.67 to 5.05 on macOS 14.5. Clearing the Cache and clicking twice on Repair fixes everything. Thanks Tony for your continued development of this exceptionally useful package!!

It doesn't help anyone updating from version 4.x, but for future versions the cache will be cleared as part of the repair process.

For anyone using a version of this project older than 5.0, you will need to need to update to the most recent version using the 'repair' function. Open the settings, click on "Repair Updater", and accept any options to download and install a working version.