This happened to work before #5516, by creating a whole shadow copy of the entire array. Revert back to that behaviour for now, it will be slow, but works still. Fixes #5590