ufcg-lsd / saps-common Goto Github PK

When performing the tests of the NFS solution in ufcg-lsd/saps-engine/issues/135, the real algorithms are executed for two tasks that are added, a LANDSAT 7 and another LANDSAT 8. The LANDSAT 8 was correctly executed, going through all the phases successfully and the results with the md5sum also expected, however, LANDSAT 7 failed (as predicted), except that the Archiver routine for failed tasks is generating an error that should occur, but not be launched, but rather be treated so that it does not loop. In this case, the error below is occurring in every routine for the same task fails for reasons that it does not have the preprocessing folder, but fails in the previous phase of inputdownloading, it should be treated, because the files in NFS temporary storage are not being deleted and this may cause disk space problems in the future.

2020-05-25 20:30:30 INFO  CatalogUtils:41 - [Retry Catalog function] Trying gets tasks with failed state using 5 seconds with time sleep
2020-05-25 20:30:30 INFO  Archiver:149 - Deleting temp data from task [fdd12b4e-cf7c-4225-bb34-1da2e6e0b85d]
2020-05-25 20:30:30 INFO  NfsPermanentStorage:82 - Archiving task [fdd12b4e-cf7c-4225-bb34-1da2e6e0b85d] to permanent storage.
2020-05-25 20:30:30 DEBUG NfsPermanentStorage:143 - Copying [/local/exports/fdd12b4e-cf7c-4225-bb34-1da2e6e0b85d/inputdownloading] into [/nfs/trash/fdd12b4e-cf7c-4225-bb34-1da2e6e0b85d]
2020-05-25 20:30:30 DEBUG NfsPermanentStorage:143 - Copying [/local/exports/fdd12b4e-cf7c-4225-bb34-1da2e6e0b85d/preprocessing] into [/nfs/trash/fdd12b4e-cf7c-4225-bb34-1da2e6e0b85d]
2020-05-25 20:30:30 ERROR Archiver:162 - Error while delete task [fdd12b4e-cf7c-4225-bb34-1da2e6e0b85d] files from disk:
java.io.FileNotFoundException: Source '/local/exports/fdd12b4e-cf7c-4225-bb34-1da2e6e0b85d/preprocessing' does not exist
        at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1300)
        at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1193)
        at org.apache.commons.io.FileUtils.copyDirectoryToDirectory(FileUtils.java:1134)
        at org.fogbowcloud.saps.engine.core.archiver.storage.nfs.NfsPermanentStorage.copyDirToDir(NfsPermanentStorage.java:144)
        at org.fogbowcloud.saps.engine.core.archiver.storage.nfs.NfsPermanentStorage.archive(NfsPermanentStorage.java:98)
        at org.fogbowcloud.saps.engine.core.archiver.Archiver.deleteTempData(Archiver.java:158)
        at java.util.ArrayList.forEach(ArrayList.java:1257)
        at org.fogbowcloud.saps.engine.core.archiver.Archiver.gc(Archiver.java:80)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Currently, there is only one configuration file for the entire component, however, the common configurations that should be separated for another file, e.g. Permanent Storage.

We are implementing a exists based on a list of a directory, but the Swift API has a more direct option.
The URL to perform the check is in this format:
https://<object_storage_address>/swift/v1/<container_name>?prefix=<task_directory>
Received 200 OK case exists the directory, otherwise 204 No Content.

The clients of this code needs to automate dependency management. We are using a container to handle this at deploy time while we are manually adding the jars at dev time.

take a look on how to manage this using maven, for example.

Currently this repository does not follow any coding style, so it was defined that it must follow the Google Java Style described here

It is necessary that the methods return a single structure for the API to return information known to the developer when using it. We have that NFS returns links from folders, however the Swift version returns links from file to file:

NFS:

[
  {
    "name": "inputdownloading",
    "url": "http://10.11.19.35:80/archiver/45b1a50e-2d2d-4f23-8fe8-33fbdbfbbe0c/inputdownloading"
  },
  {
    "name": "preprocessing",
    "url": "http://10.11.19.35:80/archiver/45b1a50e-2d2d-4f23-8fe8-33fbdbfbbe0c/preprocessing"
  },
  {
    "name": "processing",
    "url": "http://10.11.19.35:80/archiver/45b1a50e-2d2d-4f23-8fe8-33fbdbfbbe0c/processing"
  }
]

Swift:

[
  {
    "name": "preprocessing/LT05_L1TP_215065_19900125_20170131_01_T1_cpu_usage.txt",
    "url": "https://cloud5.lsd.ufcg.edu.br:8080/swift/v1/<container-name>/archiver/<id>/preprocessing/LT05_L1TP_215065_19900125_20170131_01_T1_cpu_usage.txt?temp_url_sig=2ebcf98d8fa633980c0f6b7e610b49769fb053ae&temp_url_expires=9223372036854775807&filename=LT05_L1TP_215065_19900125_20170131_01_T1_cpu_usage.txt"
  },
  ...
]

Use a Sphinx tool that generates documentation

Currently, Swift Permanent Storage considers that the Token generated by the keystone does not expire. However, the token expires and may even be invalidated in the identity service. Therefore, it is necessary to refresh the token when trying to make a request to Swift and receive the expired token response.

Swift's CLI allows you to receive credentials (projectId, userId, userPassword) and it is responsible for generating a token to perform the operation. Therefore, as a token is generated for each operation, we will never have an expired token in use.

Be based on: ufcg-lsd/saps-engine#132

As explained here, SAPS should send CPU and memory information for better job scheduling based on required resources. In this repository there is a ExecutionScriptTag class that deals with the information in the scripts and that must be modified to keep this data.