Bug in ravel_multi_index for big indexes

[cmick]

I'm working with big sparse multidimensional arrays using scipy and numpy and I've found a bug in numpy.ravel_multi_index. Specifically, for big index values (greater than 2^32) sometimes the returned value is invalid or even negative (it should be always nonnegative using the default ‘raise’ mode). Probably due to casting integers to int32 in ravel_multi_index.

Sample code that reproduces the problem:

>>> numpy.ravel_multi_index(([0], [1], [3], [19], [2379], [2], [0]), (41, 7, 120, 36, 2706, 8, 6))
array([577726140], dtype=int64)

OK

>>> numpy.ravel_multi_index(([1], [3], [3], [19], [2379], [2], [0]), (41, 7, 120, 36, 2706, 8, 6))
array([1332804284], dtype=int64)

Invalid result, should be [5627771580]

>>> numpy.ravel_multi_index(([29], [5], [117], [2], [1284], [2], [1]), (41, 7, 120, 36, 2706, 8, 6))
array([-7294480627], dtype=int64)

Invalid result, should be [117259570957]

I know that in such a case the maximum index value is huge (1,6104E+11), but not that huge.. I get the same error even when explicitly specifying all integers as numpy.int64. Setting modes {‘raise’, ‘wrap’, ‘clip’} does not have any effect either.

Tested using numpy 1.9.2 (Python 3.4.3 64bit) on Windows, and numpy 1.11.0 (Python 3.4.3 64bit) on Ubuntu.

[charris]

Hmm... I think this should not be too difficult to fix.

[jaimefrio]

There's an int s variable declared at the begining of arr_ravel_multi_index in compiled_base.c, see here, that should be a npy_intp s. We should probably also be checking for overflow, i.e. dimensions define too big an array, when using s to fill up the strides, similarly to how @seberg did in #7463 using npy_mul_with_overflow_intp.

[charris]

Still broken as of july 10, 2016.

[lagru]

This issue seems to have been fixed in #8109 and can be closed now.

[mattip]

Thanks