ValueError: could not broadcast input array from shape A into shape B

Question

I used Python with Numpy to create an array filled with numbers from a tuple, and set it up to be a 4x4 matrix.

numbers = (-8, -3, -5, 0, 4, 5, 6, 10, 7,8, 9, 100, 10, 11, 12, 1000)
numbers_array = np.array(numbers).reshape(4, 4)

Then try to insert x quantity of zeros before the last element of each row.

zeros_array = np.zeros((1,7))
numbers_final_array = np.insert(numbers_array, -1, zeros_array, 1)
print('numbers_final_array\n', numbers_final_array)

When I declare the zeros_array in that way I get the ValueError. However I the declare the zeros_array like this:

zeros_array = np.zeros((7,1))

It does work, isn't it weird? why does it happen?

I was expecting something like this at the first try.

numbers_final_array
 [[  -8   -3   -5    0    0    0    0    0    0    0    0]
 [   4    5    6    0    0    0    0    0    0    0   10]
 [   7    8    9    0    0    0    0    0    0    0  100]
 [  10   11   12    0    0    0    0    0    0    0 1000]]

Answer 1

There is a difference in the way numpy is iterating the new values to be entered into an array

nums = np.array([[1, 2, 3], [4, 5, 6]])
nums

output:

array([[1, 2, 3],
       [4, 5, 6]])

Option 1, iterate for each row in the nums array the first col new value and then the second column new value

np.insert(nums, -1, np.array([[7, 8], [9, 0]]), 1)

output:

array([[1, 2, 7, 9, 3],
       [4, 5, 8, 0, 6]])

Option 2_a, iterate through the columns in list for every row and insert all the values in the given new data array

np.insert(nums, [-1], np.array([[7, 8], [9, 0]]), 1)

output:

array([[1, 2, 7, 8, 3],
       [4, 5, 9, 0, 6]])

Option 2_b

np.insert(nums, [-1, -2], np.array([[7, 8], [9, 0]]), 1)

output:

array([[1, 8, 2, 7, 3],
       [4, 0, 5, 9, 6]])

So, for your problem:

zeros_array = np.zeros((1, 7))
numbers_final_array = np.insert(numbers_array, [-1], zeros_array, 1)
print('numbers_final_array\n', numbers_final_array)

output:

numbers_final_array
 [[  -8   -3   -5    0    0    0    0    0    0    0    0]
 [   4    5    6    0    0    0    0    0    0    0   10]
 [   7    8    9    0    0    0    0    0    0    0  100]
 [  10   11   12    0    0    0    0    0    0    0 1000]]

Answer 2

You are trying to put a (1,n) array into a slot (before the last column) of a (m,p) array. Imaging that, I think, that the 1 will broadcast to m. But:

In [485]: np.insert(np.ones((2,3),int), -1, np.zeros((1,3),int),1)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[485], line 1
----> 1 np.insert(np.ones((2,3),int), -1, np.zeros((1,3),int),1)

File <__array_function__ internals>:200, in insert(*args, **kwargs)

File ~\miniconda3\lib\site-packages\numpy\lib\function_base.py:5406, in insert(arr, obj, values, axis)
   5404 new[tuple(slobj)] = arr[tuple(slobj)]
   5405 slobj[axis] = slice(index, index+numnew)
-> 5406 new[tuple(slobj)] = values
   5407 slobj[axis] = slice(index+numnew, None)
   5408 slobj2 = [slice(None)] * ndim

ValueError: could not broadcast input array from shape (3,1) into shape (2,1)

There are two issues here. Why is the target slot (2,1), and why was the original (1,3) transposed?

In the docs there's note saying that defining the slot with -1 is different from [-1]. And an example.

>>> np.array_equal(np.insert(a, 1, [1, 2, 3], axis=1),
               np.insert(a, [1], [[1],[2],[3]], axis=1))

Indexing a (2,3) with [:,-1] will produce a (2,). Indexing with [:,[-1]] produces a (2,1).

In the insert code, there's this block:

    if indices.ndim == 0:
        # broadcasting is very different here, since a[:,0,:] = ... behaves
        # very different from a[:,[0],:] = ...! This changes values so that
        # it works likes the second case. (here a[:,0:1,:])
        values = np.moveaxis(values, 0, axis)

In your case, with axis=1, it effectively does values.T transpose.

In your case insert has to, not only determine where to insert the columns, but how many.

With a sequence obj, it can determine the slot size either from the index sequence, or the value array, or some broadcasted combination:

In [495]: np.insert(np.ones((2,3),int), [-1,-1],0,1)
Out[495]: 
array([[1, 1, 0, 0, 1],
       [1, 1, 0, 0, 1]])

In [496]: np.insert(np.ones((2,3),int), [-1],[1,2],1)
Out[496]: 
array([[1, 1, 1, 2, 1],
       [1, 1, 1, 2, 1]])

In the case of a scalar obj, the logic is different. And the code admits there are some edge cases.

I suspect it is trying to preserve this kind of case, where we insert a single column of values:

In [497]: np.insert(np.ones((2,3),int), -1,[1,2],1)
Out[497]: 
array([[1, 1, 1, 1],
       [1, 1, 2, 1]])

In [498]: np.insert(np.ones((2,3),int), -1,[[1,2]],1)
Out[498]: 
array([[1, 1, 1, 1],
       [1, 1, 2, 1]])

In [499]: np.insert(np.ones((2,3),int), -1,[[1],[2]],1)
Out[499]: 
array([[1, 1, 1, 2, 1],
       [1, 1, 1, 2, 1]])

So (2,) and (1,2) both insert a single column, but a (2,1) inserts a (2,2). It appears that it's doing the value.T so the (1,2) behaves the same as (2,).

Comparing the (1,3) with a (3,1):

In [500]: np.insert(np.ones((2,3),int), -1,[[1,2,3]],1)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[500], line 1
----> 1 np.insert(np.ones((2,3),int), -1,[[1,2,3]],1)

File <__array_function__ internals>:200, in insert(*args, **kwargs)

File ~\miniconda3\lib\site-packages\numpy\lib\function_base.py:5406, in insert(arr, obj, values, axis)
   5404 new[tuple(slobj)] = arr[tuple(slobj)]
   5405 slobj[axis] = slice(index, index+numnew)
-> 5406 new[tuple(slobj)] = values
   5407 slobj[axis] = slice(index+numnew, None)
   5408 slobj2 = [slice(None)] * ndim

ValueError: could not broadcast input array from shape (3,1) into shape (2,1)

In [501]: np.insert(np.ones((2,3),int), -1,[[1],[2],[3]],1)
Out[501]: 
array([[1, 1, 1, 2, 3, 1],
       [1, 1, 1, 2, 3, 1]])

Again the transpose is evident, with an (2,3) insert.

I'll admit that the logic of the code is not intuitive. Some examples are better than others.