[Error 5] Access is denied for create_chopped_dataset due to zarr append about l5kit HOT 6 OPEN

i closed a duplicate version of this one with wrong fomart

from l5kit.

mmm..I've never experienced this issue under linux for the test.zarr (which has around the same size of validate.zarr).

i create another chunk data set without linking to any storage on harddrive and dont get this problem, thus i strongly believe it is zarr's issue of append method

Do you mean you're holding everything in RAM?

from l5kit.

i use a local object that is similar to chunk dataset but no link to any storage in zarr_scenes_chop (zarr_utils). After the loop of append (copy). the output_dataset (linked to a storage) append the elements of the local object accordingly (append is called only one for the output_dataset)

btw, should i better use validate.zarr or test.zarr to evaluate the model?

from l5kit.

btw, should i better use validate.zarr or test.zarr to evaluate the model?

test.zarr is used in the competition, but you don't have GT for that one stored in the zarr. So If you want to chop something and also have the GT validate.zarr is better

from l5kit.

i use a local object that is similar to chunk dataset but no link to any storage in zarr_scenes_chop (zarr_utils). After the loop of append (copy). the output_dataset (linked to a storage) append the elements of the local object accordingly (append is called only one for the output_dataset)

Can you maybe link here your code?

from l5kit.

i use a local object that is similar to chunk dataset but no link to any storage in zarr_scenes_chop (zarr_utils). After the loop of append (copy). the output_dataset (linked to a storage) append the elements of the local object accordingly (append is called only one for the output_dataset)

Can you maybe link here your code?

in zarr_utils.py

    for idx in tqdm(range(len(input_dataset.scenes)), desc="copying"):
        # get data and immediately chop frames, agents and traffic lights
        scene = input_dataset.scenes[idx]
        first_frame_idx = scene["frame_index_interval"][0]
        frames = input_dataset.frames[first_frame_idx : first_frame_idx + num_frames_to_copy]
        agents = input_dataset.agents[get_agents_slice_from_frames(*frames[[0, -1]])]
        tl_faces = input_dataset.tl_faces[get_tl_faces_slice_from_frames(*frames[[0, -1]])]
        # reset interval relative to our output (subtract current history and add output history)
        scene["frame_index_interval"][0] = cur_frame_idx
        scene["frame_index_interval"][1] = cur_frame_idx + num_frames_to_copy  # address for less frames
        frames["agent_index_interval"] += cur_agent_idx - frames[0]["agent_index_interval"][0]
        frames["traffic_light_faces_index_interval"] += (
            cur_tl_face_idx - frames[0]["traffic_light_faces_index_interval"][0]
        )
        temp_dataset.scenes.append(scene[None, ...])
        temp_dataset.frames.append(frames)
        temp_dataset.agents.append(agents) 
        temp_dataset.tl_faces.append(tl_faces)
        cur_scene_idx += len(scene)
        cur_frame_idx += len(frames)
        cur_agent_idx += len(agents)
        cur_tl_face_idx += len(tl_faces)    
    output_dataset.scenes.append(temp_dataset.scenes)
    output_dataset.frames.append(temp_dataset.frames)   
    output_dataset.agents.append(temp_dataset.agents)
    output_dataset.tl_faces.append(temp_dataset.tl_faces)

output_dataset is the one linked to the storage on harddrive and temp_dataset's constructor is almost similar to chunk dataset, only a tiny difference here in zarr_dataset.py in

def initialize(
self, mode: str = "w", num_scenes: int = 0, num_frames: int = 0, num_agents: int = 0, num_tl_faces: int = 0
) -> "ChunkedDataset":
instead of self.root = zarr.open_group(self.path, mode=mode)it isself.root = zarr.open_group( 'w') # not linked to any storage`

from l5kit.