Myujth

I have an array that is not monotonic increasing. I would like to make it monotonic increasing applying a constant rate when the array decreases.

I have create a small example here where the rate is 0.2:

# Rate
rate = 0.2

# Array to interpolate
arr1 = np.array([0,1,2,3,4,4,4,3,2,2.5,3.5,5.2,7,10,9.5,np.nan,np.nan,np.nan,11.2, 11.4, 12,10,9,9.5,10.2,10.5,10.8,12,12.5,15],dtype=float)

# Line with constant rate at first monotonic decrease (index 6)
xx1 = 6
xr1 = np.array(np.arange(0,arr1.shape[0]+1),dtype=float)
yr1 = rate*xr1 + (arr1[xx1]-rate*xx1)

# Line with constant rate at second monotonic decrease [index 14]
xx2 = 13
xr2 = np.array(np.arange(0,arr1.shape[0]+1),dtype=float)
yr2 = rate*xr2 + (arr1[xx2]-rate*xx2)

# Line with constant rate at second monotonic decrease [index 14]
xx3 = 20
xr3 = np.array(np.arange(0,arr1.shape[0]+1),dtype=float)
yr3 = rate*xr3 + (arr1[xx3]-rate*xx3)

plt.figure()
plt.plot(arr1,'.-',label='Original')
plt.plot(xr1,yr1,label='Const Rate line 1')
plt.plot(xr2,yr2,label='Const Rate line 2')
plt.plot(xr3,yr3,label='Const Rate line 2')
plt.legend()
plt.grid()

The "Original" array is my dataset.
The final results I would like is the blue + red-dashed line. In the figure I highlighted also the "constant rate curves".

Since I have very large arrays (millions of records), I would like to avoid for-loops over the entire array.

Thanks a lot to everybody for the help!

Here's a different option: If you are interested in plotting monotonically increasing curve from your data, then you can simply skip the unwanted points between two successive increasing points, e.g. between arr1[6] = 4 and arr1[11] = 5, by connecting them with a line.

import numpy as np
import matplotlib.pyplot as plt

arr1 = np.array([0,1,2,3,4,4,4,3,2,2.5,3.5,5.2,7,10,9.5,np.nan,np.nan,np.nan,11.2, 11.4, 12,10,9,9.5,10.2,10.5,10.8,12,12.5,15],dtype=float)

mask = (arr1 == np.maximum.accumulate(np.nan_to_num(arr1)))

x = np.arange(len(arr1))

plt.figure()
plt.plot(x, arr1,'.-',label='Original')
plt.plot(x[mask], arr1[mask], 'r-', label='Interp.') 
plt.legend()
plt.grid()

arr2 = arr1[1:] - arr1[:-1]
ind = numpy.where(arr2 < 0)[0]
for i in ind:
 arr1[i] = arr1[i - 1] + rate

You may need to replace first any numpy.nan with values, such as numpy.amin(arr1)

I would like to avoid for-loops over the entire array.

Frankly speaking, it is hard to achieve no for-loops in numpy, because numpy as the C-made-library uses for-loops implemented in C / C++. And all sorting algorithm (like np.argwhere, np.all, etc.) requires comparisons and therefore also iterations.

Contrarily, I suggest using at least one explicit loop made in Python (iteration is made only once):

arr0 = np.zeros_like(arr1)
num = 1
rate = .2
while(num < len(arr1)):
 if arr1[num] < arr1[num-1] or np.isnan(arr1[num]):
 start = arr1[num-1]
 while(start > arr1[num] or np.isnan(arr1[num])):
 print(arr1[num])
 arr0[num] = arr0[num-1] + rate
 num+=1
 continue
 arr0[num] = arr1[num]
 num +=1

Your problem can be expressed in one simple recursive difference equation:

y[n] = max(y[n-1] + 0.2, x[n])

So the direct Python form would be

def func(a):
 out = np.zeros_like(a)
 out[0] = a[0]
 for i in range(1, len(a)):
 out[i] = max(out[i-1] + 0.2, a[i])

 return out

Unfortunately, this equation is recursive and non-linear, so finding a vectorized algorithm may be difficult.

However, using Numba we can speed up this loop-based algorithm by a factor of 300:

fastfunc = numba.jit(func)

arr1 = np.random.rand(1000000)

%timeit func(arr1)
# 599 ms ± 13.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit fastfunc(arr1)
# 2.22 ms ± 107 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

I finally managed to do what I wanted with a while loop.

# data['myvar'] is the original dataset I want to reshape
data['myvar_corrected'] = data['myvar'].values
temp_d = data['myvar'].fillna(0).values*1.0
dtc = np.maximum.accumulate(temp_d)
data.loc[temp_d < np.maximum.accumulate(dtc),'myvar_corrected'] = float('nan')
stay_in_while = True
min_rate = 5/200000/(24*60)
idx_next = 0
while stay_in_while:
 df_temp = data.iloc[idx_next:]
 if df_tem['myvar'].isnull().sum()>0:
 idx_first_nan = df_temp.reset_index().['myvar_corrected'].isnull().argmax()

 idx_nan_or = (data_new.index.values==df_temp.index.values[idx_first_nan]).argmax()

 x = np.arange(idx_first_nan-1,df_temp.shape[0])
 y0 = df_temp.iloc[idx_first_nan-1]['myvar_corrected']
 rate_curve = min_rate*x + (y0 - min_rate*(idx_first_nan-1))

 damage_m_rate = df_temp.iloc[idx_first_nan-1:]['myvar_corrected']-rate_curve

 try:
 idx_intercept = (data_new.index.values==damage_m_rate[damage_m_rate>0].index.values[0]).argmax()
 data_new.iloc[idx_nan_or:idx_intercept]['myvar'] = rate_curve[0:(damage_m_rate.index.values==damage_m_rate[damage_m_rate>0].index.values[0]).argmax()-1]
 idx_next = idx_intercept + 1
 except:
 stay_in_while = False
 else:
 stay_in_while = False
# Finally I have my result stored in data_new['myvar']

In the following picture the result.

Thanks to everybody for the contribution!