Constrain fitting parameter to integer

I'd be curious about the application or the fitting function!

My first thought was to do a normal fit with real numbers, then round and fix the ones that are supposed to be integers as constants, then run the fit again for slight tweaking of all other parameters.

Fitting constrained to integers is REALLY problematic. The algorithms used are fundamentally based on continuous arithmetic.

Here is a way to proceed that I can't really endorse:

1) Your fitting function can (possibly) take the coefficient and round it to an integer.

2) This will result in singular matrix errors as a small perturbation of the coefficient will lead to a zero derivative. To fix that, use an epsilon wave: DisplayHelpTopic "The Epsilon Wave"

Set the epsilon for your integer coefficient to something large, like 2 or more.

3) Pray for convergence to any god that you choose :)

Possibly a better approach would be to try out Andy Nelson's GenCurveFit XOP: https://www.wavemetrics.com/project/gencurvefit

Use GenCurveFit to get a "pretty good" fit, including a possible value for the integer coefficient. This may work because GenCurveFit uses a derivative-free genetic algorithm for optimization. Once you have a "pretty good" fit, then use Igor's FuncFit to polish up the fit, but you will need to hold the integer coefficient to avoid a singular matrix.