1. INTRODUCTION

Knowledge of a camera spectral sensitivity functions is important for many applications, including multispectral imaging [1,2], spectral reflectance and illuminant estimation [3–5], color correction [6,7], and color constancy [8,9]. Since these functions are not available from the camera manufacturers, they must be measured or estimated.

In the lab, a camera sensor’s spectral sensitivities are measured by recording the device’s rgb response to light for narrowband stimuli at a range of wavelengths across the visible spectrum. The narrowband stimuli might be produced by a monochromator [10], integrating sphere [11], or sets of narrowband interference filters [12,13]. However, this lab-based method is both time-consuming and expensive to carry out, and moreover is not available to most users.

There has been a sustained interest in the digital imaging community in developing estimation methods for recovering device sensitivities. The idea is to infer the spectral shape of the r, g, and b spectral sensitivity curves given known spectral measurements of the light entering the camera and the corresponding recorded camera rgb values. Assuming the image is linear—i.e., when light is doubled the response is doubled—the estimation problem can be posed as a linear regression. Thus, we can solve for the spectral function of the sensor mechanisms over the visible spectrum [14–16].

Of course, nowadays most images—for example, from mobile phones—are not linear but rather they are rendered. The difference between raw and rendered is shown in Fig. 1. To a tolerable first approximation, raw images are mapped to rendered counterparts in two stages. First the raw rgb is mapped to a corresponding display RGB via a $3\times 3$ matrix transform. This so-called color correction transform accounts for the display and color preference [17]. The color-corrected RGB values are then mapped by an increasing function $f()$, which accounts for gamma, tonality, and preference [18–21]. In this paper, for rendered images, we are also interested in recovering the effective device spectral sensitivities, which we define to be the device sensors multiplied by the (unknown) color correction transform.

 

Fig. 1. Image of the Macbeth SG color chart. Left: raw (color corrected). Right: rendered camera output.

Download Full Size | PDF

Our method is based on rank orders of the R, G, and B pixel values in an image. Our observation is that if the response for one patch in a color target is larger than that measured for a second patch, then this places a constraint on the shape of the spectral sensitivity curve (of the sensor).

Let us illustrate this concept in a simple example. Suppose that we have two surfaces, one of which is a red patch, which is highly reflective in the red part of the visible spectrum, and the other a purple patch, which is slightly less reflective in the long wavelengths but has significant reflectivity in the shorter wavelengths. Now, we measure the red reflectance (illuminated by white light) with a red-sensitive sensor. The red patch response is larger than that recorded for the purple patch. This makes sense, as the red sensor is most sensitive to longer wavelengths. In contrast, if the sensor were sensitive only to short wavelengths, then we would record an opposite rank order, i.e., the response to the purple reflectance would be greater than for the red one. Thus, by looking at a single rank ordering of two reflectances we can immediately distinguish—albeit in coarse terms—whether a sensor is more likely to be long-wave or short-wave sensitive.

Of course, we actually know whether we are trying to calibrate a red or blue sensor but this rank ordering idea turns out to be very powerful in determining the shape of the underling spectral sensitivity curve (of the sensor). We show that each pair-wise rank ordering splits the space of all spectral sensitivities into two parts: a feasible region and an infeasible region. The intersection of feasible regions (computed for pairs of responses) delimits a feasible region of sensor space from which we pick our overall sensor estimate.

By construction, our Rank-based method is invariant to any increasing function that is applied to the raw rgb values of an image (since the application of an increasing function will not change the pair-wise rank orders). It follows then that the Rank-based method applied to rendered images can recover the effective device linear spectral sensitivities (the actual raw spectral sensor sensitivities multiplied by the color correction matrix).

We benchmark the spectral sensitivities recovered by different algorithms against the spectral sensitivity functions of Nikon D5100 and Sigma SD1Merill cameras that we measured at the National Physical Laboratory (NPL) [10]. When the camera response is linear, we demonstrate that the proposed Rank-based method provides estimated spectral sensitivity functions that are equally accurate as those returned by cited antecedent spectral sensitivity estimation methods. However, when camera response is nonlinear, Rank-based spectral estimation is shown to provide a step change in our ability to estimate the effective camera spectral sensitivities.

In Section 2, we review color image formation along with prior art camera spectral sensitivity estimation algorithms. Rank-based spectral sensitivity estimation is described in Section 3. We evaluate our method on linear and rendered images in Section 4. The paper concludes in Section 5.

2. BACKGROUND

The camera response at the $j$th pixel from the $i$th sensor can be modeled as

We combine the light multiplied by reflectance into a single spectral quantity, the color signal, denoted by $C(\lambda )=E(\lambda )S(\lambda )$. In the discrete representation where spectral quantities are represented by vectors, the color signal for the $j$th reflectance, ${\underset{\_}{c}}^j$, is computed as

Let $C$ denote the $n\times N$ matrix of color signal spectra (one per row). The corresponding $n$ responses are written as

In spectral sensitivity estimation, it is assumed that the color signal matrix $C$ and the camera responses are known. This restriction is not overly onerous since we might probably know the spectral reflectance in a color target and we can calibrate our camera under a light source. We can solve for the sensor spectral sensitivity by minimizing:

The least-squares closed-form solution to Eq. (7) is written as

 

Fig. 2. Top: spectral response function of Nikon camera measured at NPL [10]. Bottom: jaggy (high norm) spectral response function estimated using Eq. (7) on 140 reflectance patches of Xrite SG chart.

Download Full Size | PDF

At the heart of the Moore–Penrose pseudoinverse is the raw cross product $C^{t}C$. This matrix is only stably invertible if it has full rank (the jaggy recovery shown in Fig. 2, in essence, informs us that the matrix does not meet this condition). The raw cross-product matrix is full rank if and only if the set of color signal spectra spans the set of all possible spectra. In practice, this is unlikely to be the case, since surface reflectance is inherently smooth [24]. Parkkinen et al. [25], among others [13,26], estimate that the dimensionality of spectral reflectance is between 6 and 9. When the raw cross-product matrix has low rank this means that very small perturbations to the measurements (in this case $\underset{\_}{P}$) result in very different regression results. Both the ground-truth and jaggy sensors in Fig. 2 describe the data very well. There exists a small perturbation to $\underset{\_}{P}$ that would result in the correct ground-truth sensors being recovered.

A specific and interesting condition, which immediately forces $C^{t}C$ to be rank deficient, is when the number of measurements is less than the dimensionality of the matrix. When we have fewer known variables than unknowns, there are (infinitely) many sensors that exactly account for the data. A classic approach is to find the “least-norm” sensor from among these sensors. Suppose we assume that the sensor $\underset{\_}{q}$ is in the row space of $C$, i.e., $\underset{\_}{q}=C^t\underset{\_}{c}$. Clearly, one solution for $\underset{\_}{c}$ would be

In other words, the sensor is defined as $\underset{\_}{q}=C^t{(C{C}^t)}^{-1}\underset{\_}{P}$. It is straightforward to show that this solution, compared against all others, has least-norm [27]. This idea of solving for a sensor while implementing a secondary constraint, in the case of least-norm, is often useful in spectral sensitivity estimation. Referring back to Fig. 2, the norm (magnitude) of the actual sensor is much less than the simple least-squares estimation since the more jaggy the sensor the higher its norm. This idea is exploited by the prior art estimation algorithms.

A. Prior Art Estimation Algorithms

1. Regularization

Equation (7) is regularized by adding a penalty term on the norm,

The regularized least-squares solution to spectral sensitivity recovery is written in closed form as

A natural question arising upon the application of either Regularization method is how to set $\gamma$. A good solution is one that makes both the least-squares error and the norm simultaneously small. The best compromise solution is usually determined by trying a variety of different $\gamma$ values and then applying a statistical criterion to choose the best gamma overall [30].

2. Subspace Methods

Suppose, the spectral sensitivity of sensors is described by a low-dimensional linear model,

The least-squares solution of Eq. (14) is

In [11,31] linear models were derived by the principal component analysis (PCA) of measured spectral sensitivities of 28 different Nikon and Canon camera models. Remarkably, it was found that even using just the first two principal components of this dataset, it was sufficient to account for their data. Zhao et al. [32] proposed using a simple Sine basis expansion, where between 7 and 10 basis functions were used.

3. Convex Optimization

Equation (10) is constrained by incorporating soft penalties into the problem formulation. However, there are also strong constraints which we might usefully apply. For example, we know that the spectral sensitivities of the devices are zero at both ends of the visible spectrum and also they tend to have one, or at most two peaks. These “harder” constraints are often described using linear equalities or inequalities. Optimizing a linear objective such as Eq. (6) subject to linear equality/inequality constraints is a convex optimization.

In [33] sensor estimation is posed as minimizing a linear objective function subject to linear constraints and the sensors are found using linear programming. Finlayson et al. [14] proposed that assuming a known peak sensor response at the $z$th sample point, the spectral sensitivity is estimated as

The standard Fourier basis is used for $B$, which up to a given band-limited frequency is composed of Sine and Cosine basic functions. The formulation in Eq. (16) is a quadratic programming problem (quadratic objective with linear constraints). The global best solution for quadratic programming and linear programming problems can be solved by efficient simplex-type search algorithms.

B. Measured Spectral Sensitivity Functions

In this paper, we use the measured spectral sensitivities of Nikon D5100 and Sigma SD1Merrill cameras as the benchmark to evaluate spectral estimation algorithms. The cameras were rigorously measured at the National Physical Laboratory [10]. The resulting measured spectral response functions for both Nikon and Sigma cameras are illustrated in Fig. 3 and are also accessible online [34].

 

Fig. 3. Measured Nikon (top) and Sigma (bottom) camera sensitivities from NPL calibration facility [10,34].

Download Full Size | PDF

3. RANK-BASED METHOD

In this section, we show how we can use a single image of a color chart to estimate the camera spectral sensor sensitivities irrespective of whether the image is linear or rendered (nonlinear). That we can solve the latter case is an improvement over all the prior art.

A. Rank-Based Spectral Estimation

The rank order of two single-channel linear responses $p_a$ and $p_b$ is encapsulated by writing an inequality:

We note that, if we apply a monotonically increasing function $f()$ to both sides of the above inequality, the rank order will still be preserved:

Let us revisit our vector-based model of image formation, Eq. (5), and write the pair-wise rank order of two responses,

Equation (20) has a useful geometric interpretation. It teaches us that $\underset{\_}{q}$ must belong in one half of the space of all sensors. In other words, if $\underset{\_}{q}$ satisfies the inequality, then $-\underset{\_}{q}$ does not. Geometrically, Eq. (20) defines a half-space constraint. We denote this half-space as $H({\underset{\_}{d}}^j)$. The intersection of all the half-spaces delimits an unbounded feasible convex region of sensor space. According to rank orderings, the actual true spectral sensitivity must lie in this feasible set. Mathematically, we write

Let us visualize Eq. (20), by assuming the (grossly over) simplified problem, where we seek only to estimate spectral sensitivities at two wavelengths. With respect to this simplification, each half-plane constraint of Eq. (20) divides the Cartesian plane into two halves. In Fig. 4 we show four half-plane constraints where solid arrows show which side of the line is the feasible half-space. The overall bounded feasible convex cone is the hatched area shown. In this paper, we are assuming that our spectral sensitivities are 31-vectors (the value of the sensor at 10 nm sampling intervals for the visible spectrum of 400 to 700 nm). Thus, the feasible set is a 31-dimensional unbounded convex cone. The unbounded cone encapsulates the idea that with rank ordering constraints alone, we can constrain the shape (or, vector orientation) but not its scale (not the vector magnitude).

 

Fig. 4. Illustration of the intersection of four half-planes with blue diagonal stripes.

Download Full Size | PDF

B. Rank-Based Method for Rendered Images

To a tolerable approximation [18] raw rgb values are mapped to rendered counterparts in two steps. First, the raw rgb values are mapped by a $3\times 3$ color correction matrix $M$ (which accounts for the display primaries [35] and preference [17]), resulting in RGB values. The effective sensitivities for a rendered image, $\mathcal{Q}$, are defined as

In a second step, an overall, or per channel, transfer function $f()$ is applied to the color-corrected RGB values. Every transfer function is increasing and implements display gamma and tone adjustment [17]. The mathematical model for a rendered image is written as

Note, we drop $f()$ because this function is increasing (does not change the ranks of the responses). For rendered images we can only recover the effective sensitivity ${\underset{\_}{\mathcal{Q}}}^i$.

C. Implementation

In spectral sensitivity estimation, in general, we attempt to recover the spectral sensitivities of a camera by taking a single picture of a color target (e.g., see Fig. 1) where we assume that the reflectances of the color target are known and that the prevailing illuminant spectral power distribution is measured. In Rank-based estimation, each pair of responses (per channel) can be related to a color difference signal and a linear inequality of the form in Eq. (19). Taken together the set of inequalities delimits an overall feasible region of sensor space where the spectral sensitivity must be. To choose a single member from this set, we seek a sensor that has zero response at both ends of the visible spectrum and integrates to 1. Also inspired by the work of Zhao et al. [32], we seek for a sensor that spans a seven-dimensional Sine basis. By these, we assure that the chosen sensor has the commonly addressed usual characteristics of spectral sensor sensitivity functions. The Rank-based spectral sensitivity is recovered by minimizing the following quadratic program,

D. Cautionary Remark

Potentially, the rendering Eq. (23) is actually yet more complex [18]. In particular, it is often the case that cameras implement a gamut mapping step which we have not included in our model. The need for gamut mapping is due to the fact that the color correction matrix $M$ will map raw image colors outside the gamut, which cannot be displayed. Assuming display RGB is the unit cube (all values between 0 and 1), some of the mapped colors might be negative or greater than 1. Gamut mapping [36] moves colors, in a proprietary nonlinear way, toward the neutral axis so that all the colors in an image are displayable, i.e., all colors lie in the unit cube. Denoting the gamut mapping function as $\mathrm{\Gamma }()$, rendered image formation is written as

However, in rendering, the color correction matrix and per channel functions $f_i()$ provide most of the heavy lifting and in fact, if the gamut mapping step is left out, the resulting image is a tolerable approximation of the gamut-mapped version. This is especially true for image colors that are not saturated (as the effect of any gamut mapping step can be ignored for colors that lie near the neutral axis).

Empirically, we found that the gamut mapping step did not change the rank order of the responses. But, equally, we found that restricting ourselves to less saturated colors still provided a strong Rank-based constraint and our method continued to work well.

4. EXPERIMENTS AND RESULTS

A. Data Acquisition

We took an image of a 140 color chip Xrite SG color checker chart using the Nikon D5100 and Sigma SD1Merril cameras. This will give us 9730 color pairs. The chart is imaged in a VeriVide cabinet, facing upward toward the D65 light source at an angle of approximately 45 deg. The $P_{140\times 3}$ matrix of rgb values is the average camera response for the central $10\times 10\text{pixel}$ region of each checker patch. We made sure that the images are nonclipped, by adjusting the exposure level of the camera. The dark current noise level is measured (by taking a picture with the lens cap on) and is subtracted from the raw image.

A PR670 spectroradiometer was, in analogy to the camera capture conditions, positioned approximately at 45 deg relative to the color checker facing the VeriVide cabinet. The spectroradiometer—which measures the spectrum of light at a single spot in the image—was held fixed. That is, the color checker was moved so that each color patch was measured at the same location (with respect to the same 45 deg viewing geometry). The per-chip spectral measurements comprise the matrix $C_{140\times N}$ (there are 140 chips in the color checker and we measure the visible spectrum, 400 to 700 nm, at a 10 nm sample interval).

We benchmark spectral sensitivity estimation against measurements made at the National Physical Laboratory presented in Section 2.B. The effective sensitivities—see Section 3.B for a discussion—of the Nikon and Sigma cameras are found via the radiometric calibration method [18]. We test three cited antecedent methods: Tikhonov-based on derivatives [29], the subspace method [11] (using just 2 basis functions per sensor class), and quadratic programming [14]. In the data tables that follow these methods are, respectively, denoted “Regularization,” “subspace,” and “QP.” Finally, we compute recovery statistics for our new method “Rank-based.”

B. Error Metrics

1. Percentage Spectral Recovery Error

Let $\underset{\_}{q}$ and $\underset{\_}{\hat{q}}$ denote the actual and estimated sensor. The percentage spectral recovery error is calculated as

Although intuitive, the percentage spectral error tends to be sensitive to small spectral shifts in the position of the estimated sensor. The spectral error between a sensor and a duplicate that is shifted 10 nm toward the longer wavelengths can be very large despite the fact that both sensors have an integrated response that is similar. This discrepancy has led researchers to measure the goodness of a recovery in terms of how well they predict actual sensor values.

2. Vora Value Metric

This metric was designed to compare two sensors in terms of how similar they integrate all theoretically possible spectra [12,21]. Suppose that ${\underset{\_}{p}}_i$ and ${\underset{\_}{\hat{p}}}_i$ denote the actual and predicted rgb camera response. The percentage response error (RE) is calculated:

A key insight of Vora is that we might be able to reduce the RE errors by color correcting the measured rgb values,

3. CIE Lab $\mathrm{\Delta }\mathit{E}$ Error Metric

The CIE Lab formula provides a measure for how close two reflectances appear to a human observer under a known illuminant. The error metric is designed for the XYZ color space (which to a good approximation is a linear transform from the cone sensitivities in the eye). To calculate the $\mathrm{\Delta }E$ error, the three vectors of XYZ responses are measured from two surfaces and then converted to corresponding lab triplets. The Euclidean distance between these triples roughly correlates with our perception of color difference with a $\mathrm{\Delta }E$ error of one signifying, on average, just a noticeable difference [37].

Before we can use the metric for camera sensitivities we need to—via a calibration procedure—best map the camera rgb responses to corresponding XYZ coordinates and then to lab values. We adopt the methodology defined in [38]. Taking the 1995 reflectances measured in [39] and CIE D65 illumination we numerically calculate using Eq. (2) the rgb values of actual versus estimated rgb values and transform these into lab errors. The mean, median, and 95% quantile errors are then used to assess how well we have estimated the camera sensors.

C. Spectral Estimation for Raw Images

In Fig. 5, the ground-truth spectral sensor sensitivity functions [10] from Nikon (top) and Sigma (bottom) are plotted in solid lines. The recovered sensors using our Rank-based method are plotted with dashed lines. All sensors are scaled to have a maximum sensitivity of one. Overall curve (of the sensor) of the estimated spectral sensitivity curve (of the sensor) are similar to the NPL ground-truth data throughout the spectrum.

 

Fig. 5. Solid lines, the Nikon D5100 (top) and Sigma spectral sensor sensitivities and estimated Nikon spectral sensor sensitivities (bottom). Dashed lines are the sensors recovered by Rank-based spectral estimation.

Download Full Size | PDF

We evaluate the sensitivity estimation returned by the Rank-based method with the prior art in Table 1. The first five columns of this table show, respectively, the spectral recovery error for the R, G, and B color channels; the average spectral recovery over all channels; and the Vora value. Boldface numbers denote the best statistic for each column. The method with the smallest statistic is the subspace method of [11]. This method uses Eq. (14) where a two-dimensional sensor basis is used (derived by a PCA analysis of spectral sensitivity functions of 28 cameras). By chance the measured Nikon camera is very well modeled by this basis set.

Table 1. Percentage Spectral Error of Each Channel, the Average Across All Channels, and the Vora Values Are Shown for Raw Images from Nikon (First Five Columns) and Sigma (Last Five Columns) Cameras

View Table | View all tables in this article

However, we see that all methods have a low Vora value indicating that the rgb values modeled by the recovered sensors are close to those of ground-truth sensors. Encouragingly, the average spectral recovery method of the Rank-based method (excluding the 2D subspace method) is similar to the prior art. That is, a method that uses only the rank orders of the responses works equally well as methods that fit the recorded data (we achieve the same performance using fewer of the degrees of freedom in the data).

In columns 6 to 10 of Table 1, we summarize the recovery performance for the Sigma Merill camera. Again the Rank-based method works well and performs similar to quadratic programming in terms of the Vora value. Notice that for this dataset the subspace method works least well (because the Sigma sensitivities are not in the assumed subspace).

Table 2 reports the CIE Lab error. Again the Rank-based method delivers good performance. Broadly, it supports performance similar to the prior art.

Table 2. CIE Lab $\mathsf{\Delta }\mathsf{E}$ Errors between the Predicted and Estimated Camera Responses to the 1995 Sample Reflectances [39] under the D65 Light [38] from Nikon (First Three Columns) and Sigma (Last Three Columns) Cameras

View Table | View all tables in this article

D. Spectral Estimation for Rendered Images

As described in Section 3.B, a rendered image (a typical JPEG from a smart phone) is nonlinear. To a tolerable approximation the raw image colors are mapped by a $3\times 3$ color correction matrix and then tone-curves are applied to the color-corrected rgb values to make the final image (see Fig. 1 for a raw and rendered pair). Because the Rank-based method is based on rank orders only, it should be able to recover the effective sensitivities of a rendered image. Effective sensitivities are the camera sensors multiplied by the color correction matrix.

To evaluate how well we can recover the effective sensitivities we use the rendered image of the SG Macbeth—shown in the right of Fig. 1—color chart as an input to the Rank-based method resulting in the estimated sensors illustrated with dashed lines in Fig. 6. Here the ground-truth spectral sensitivity functions (solid lines) are the NPL sensors [10], multiplied by a color correction matrix $M$, which was found using a calibration procedure described in [18]. Visually, the Rank-based method is able to recover the effective sensitivities of our two cameras.

 

Fig. 6. Solid lines, the Nikon D5100 (top) and Sigma spectral sensitivities and estimated Nikon spectral sensor sensitivities (bottom). Dashed lines are the sensors recovered by Rank-based spectral estimation using rendered image.

Download Full Size | PDF

Table 3 reports the error metrics (in the same format of Table 1). For both Nikon and Sigma cameras, we observe high spectral recovery errors and Vora values for all the three cited antecedent methods. This illustrates their inability to recover effective sensitivities in the face of the nonlinearity present in a rendered image. However, the Rank-based method continues to work well. The CIE Lab errors reported in Table 4 reaffirm this result. Rank-based delivers good estimation. In contradiction, all the prior art is unable to accurately model color outputs when the calibration data is nonlinear.

Table 3. Percentage Spectral Error of Each Channel, the Average Across All Channels, and the Vora Values Are Shown for Rendered Images from Nikon (First Five Columns) and Sigma (Last Five Columns) Cameras

View Table | View all tables in this article

Table 4. CIE Lab $\mathsf{\Delta }\mathsf{E}$ Errors between the Predicted and Estimated Color-Corrected Camera Responses to the 1995 Sample Reflectances [39] under the D65 Light [38] from Nikon (First Three Columns) and Sigma (Last Three Columns) Cameras

View Table | View all tables in this article

5. RANK-BASED VERSUS LINEARIZATION

The reader might wonder whether the poor performance of the prior art methods working on rendered data might be mitigated by linearizing the data first. We test this idea here. We consider two types of linearization. First, we assume that a rendered image might be made approximately linear by undoing the gamma (e.g., undoing the sRGB gamma [35], roughly raising the image to the power 2.2). Second, we use the achromatic colors in a color checker—here the six patches in the central part of the SG color checker—to linearize the data. With respect to the second method, the idea is to plot the camera responses against the % spectral reflectance for the different gray reflectances. Using linear interpolation we can define a function that, at least approximately, linearizes the data (for more discussion, see [40]).

In Table 5 we show the recovery errors for the prior art methods for the simple gamma-only linearization. It is clear this method is not sufficient for linearizing the data (the sensor spectral estimation is poor). In Table 6, the six patch linearization is shown. While the sensor spectral sensor estimation is improved, the recovery still has large errors. Neither linearization recovers the effective sensitivities as well as the Rank-based method (last row of Table 3).

Table 5. Percentage Spectral Error of Each Channel, the Average Across All Channels, and the Vora Values Are Shown for Rendered Images Linearized by Gamma Correction from Nikon (First Five Columns) and Sigma (Last Five Columns) Cameras

View Table | View all tables in this article

Table 6. Percentage Spectral Error of Each Channel, the Average Across All Channels, and the Vora Values Are Shown for Rendered Images Linearized by Interpolation from Nikon (First Five Columns) and Sigma (Last Five Columns) Cameras

View Table | View all tables in this article

Initially, we were surprised that linearizing the data with the achromatic scale led to such poor sensor estimation. After all, the prior art does work tolerably well on actual linear data. Our results would suggest that a small departure from linearity would hamper the prior art from recovering the sensors. To test this idea we wished to measure how well a method recovers the spectral sensitivities of a camera as a function of increasing nonlinearity.

Using our raw data as a starting point we made the data slightly nonlinear by raising it to the power $\rho$. When $\rho =1$ our data is linear and the recovery results shown in Table 1 hold. Now, we can increase or decrease the power, $\rho =1.1$ or $\rho =0.9$, and in either case the data remains quite linear. Even larger or even smaller powers ($\rho =2$ or $\rho =0.5$) make the data progressively nonlinear. In the left panel of Fig. 7, for each of the three prior art methods, we plot the Vora value of the recovered sensitivities as a function of the $\rho$ power. It is immediately clear how sensitive the prior art is to the linearity of the data. Even small departures from linearity have a marked effect on the accuracy of the spectral sensitivity estimation. In the right of Fig. 7 we plot the same result as a function of average spectral estimation error.

 

Fig. 7. Plot of Vora (left) and averaged spectral error (right) calculated for Nikon raw image when raised to power values indicated on the x-axis.

Download Full Size | PDF

For each of the prior art methods, even very small nonlinearities can lead to a doubling in recovery error. Of course, the Rank-based method continues to work well in the face of small and large nonlinearities. For the nonlinearities there is no change in the Rank-based estimated sensors.

6. CONCLUSION

In this paper, we have shown how the rank order of camera responses is a powerful tool for estimating the camera spectral sensitivity functions. We have described how the rank orders of image rgb values impose half-space constraints as ranked pair splits the space where the spectral sensor sensitivity must lie in one of the two parts. A plausible sensor estimate lies in this region where all half-space constraints intersect, which can be found using constrained quadratic programming. The Rank-based method works well on linear raw (as well as prior art), but delivers a step change in performance given nonlinear data. A key example of the latter is rendered images (e.g., normal JPEGs) where we show we can deliver the estimated sensitivities of a camera. In contradiction, the prior art does not work for rendered data even when it is linearized.