Uploading and Managing Files
Uploading a file to dStorage
./zbox upload
command is used to upload a file on dStorage. Parameters for uploading a file to dStorage can be viewed by using the ./zbox upload --help command.
Here are the Parameters :
Parameter | Required | Description | Default | Valid values |
---|---|---|---|---|
allocation | yes | allocation id, sender must be allocation owner | string | |
commit | no | save metadata to blockchain | false | boolean |
encrypt | no | encrypt file before upload | false | boolean |
localpath | yes | local path of the file to upload | file path | |
remotepath | yes | remote path to upload file to, use to access file later | string | |
thumbnailpath | no | local path of thumbnail | file path |
Example :
Let's upload an info.txt file to allocation, the local path for the file is mapped to a local
variable and the remote directory is mapped to a remote
variable which is myfiles/info.txt.
Sample Command:
Response:
Verify whether the uploaded file is available on dStorage using list command required parameters are allocation ID and remote path
Response:
The response yields a lookuphash which works as a unique identifier for the uploaded file. Since you uploaded the file, the download payer section will show it as owner
.
Copy and mark your lookup hash to a variable as it will be used for performing other file operations
export LOOKUPHASH=f15383a1130bd2fae1e52a7a15c432269eeb7def555f1f8b9b9a28bd9611362c
Uploading an Encrypted File to dStorage
Parameters for encrypting and uploading a file to dStorage are almost the same as uploading a file on dStorage and only requires an additional --encrypt
flag with the command.
For instance if we want to encrypt and upload a sample info.txt
file the command would be:
Command:
Response:
To verify whether the file is encrypted and uploaded to dstorage use the list command :
Response:
Scroll right in the response and check the IS ENCRYPTED
section, it will say YES
if the file is encrypted.
Updating the contents of a file on dStorage
Use ./zbox update
command to update the content of an existing file in the remote path. Like upload command. Only the owner of the allocation or a collaborator can update a file. To add collaborators to an allocation, check File Collaboration.
Parameters for updating the contents of the file on dStorage can be seen by typing ./zbox update --help
Here are the parameters:
Parameter | Required | Description | Default | Valid values |
---|---|---|---|---|
allocation | yes | allocation id | string | |
encrypt | no | encrypt file before upload | false | boolean |
localpath | yes | local file to upload | file path | |
remotepath | yes | remote file to upload | string | |
thumbnailpath | no | local fumbnail file to upload | file path | |
commit | no | save meta data to blockchain | false | boolean |
Sample Command:
Verify the contents of updated file on dStorage using the list command.
Sample Response:
Create Directory
./zbox createdir
command is used to create directory on allocation for storing files.
Parameter | Description | Valid Values |
---|---|---|
--allocation | Provide Allocation ID | string |
--dirname | Provide Directory Name and absolute path . | string |
--h,--help | help for createdir | int |
Sample Command:
Sample Response:
Note: To verify whether directory is created successfully run List all files command.
Sharing a File on dStorage
Zbox supports sharing of files using auth tokens. Sharing can be encrypted or not encrypted for this section we will describe the non-encrypted share functionality. For encrypted shared functionality have a look at Private Sharing .
./zbox share
command is used to generate an authtoken that provides authorization to the holder to the specified file on the remotepath. Parameters for sharing the file can be viewed using the ./zbox share --help
The parameters are:
Parameter | Required | Description | Valid values |
---|---|---|---|
allocation | yes | allocation id | string |
clientid | no | id of user to share file with, leave blank for public share | string |
encryptionpublickey | no | public key of the client to share file with, required if clientId | string |
expiration-seconds | no | seconds before | int |
remotepath | yes | remote path of file to share | string |
revoke | no | revoke share for remote path | flag |
Command:
The response will yield an Auth token which can be shared publicly to users for downloading the file.
Sample Response:
Note: Use an export command and mark your Auth token to a variable to use with more commands
Downloading the file from dStorage
The ./zbox download
can be used to download the file in two ways from dStorage. Using the Allocation ID or Authticket.
Download using Allocation ID and Remotepath
Parameters for downloading the contents of file from dStorage can be viewed using the./zbox download --help
Here are the parameters:
Parameter | Required | Description | Default | Valid values |
---|---|---|---|---|
allocation | yes | allocation id | string | |
authticket | no | auth ticked if not owner of the allocation, use share to get auth ticket | string | |
blockspermarker | no | download multiple blocks per marker | 10 | int |
commit | no | save metadata to blockchain | false | boolean |
endblock | no | download until specified block number | int | |
localpath | yes | local path to which to download the file to | file path | |
remotepath | yes | remote path to which the file was uploaded | string | |
rx_pay | no |
| false | boolean |
startblock | no | start download from specified block | int | |
thumbail | no | only download the thumbnail | false | boolean |
Command:
Sample Response:
If you are not able to download files and getting Error in file operation: File content didn't match with uploaded file response check Error 2 in Tips and Troubleshooting Section.
Download using Authticket
The necessary flags for downloading the file using the Authticket is auth ticket and local path. Get the AuthToken for the file on dstorage from another user and mark it to a variable. We will use auth token created and marked to AUTH
variable for the file we shared in sharing a file on dStorage section .
Command:
Response:
As you can see Zbox has identified the file(info.txt) based on the auth token and didn't require any remote path and allocation id.
Deleting a File from dStorage
Deleting the file on dStorage can be done using the ./zbox delete
command. Parameters for deleting the contents of the file on dStorage can be seen by typing ./zbox delete --help
Here are the parameters:
Parameter | Required | Description | Default | Valid values |
---|---|---|---|---|
allocation | yes | allocation id | string | |
remotepath | yes | remote path of file to delete | string | |
commit | no | save meta data to blockchain | false | boolean |
Command:
Sample Response:
Verify whether the updated txt file is available on dstorage using the list command
Response:
As you can see the file is not showing in remote directory which means the file has been successfully deleted.
Copy File to Another Existing Folder in dStorage
Use ./zbox copy
command to copy file to another folder path in dStorage. Only the owner of the allocation can copy an object.
By using ./zbox copy --help
.You can have a look at the parameters:
Parameter | Required | Description | default | Valid values |
---|---|---|---|---|
allocation | yes | allocation id | string | |
commit | no | save metadata to blockchain | false | boolean |
remotepath | yes | remote path of object to copy | string | |
destpath | yes | destination, an existing directory to copy object | string |
Required Flags for copying the file to another folder are Allocation ID, the remote path to the existing file on dstorage, and the new destination path/folder path to copy a file to.
Steps :
Verify whether the sync.txt file is available in the remote path using list command.
Response:
2. Use the ./zbox copy
command to copy files.
Successful Response:
3. Verify whether the file has copied using:
Response:
Moving Files Between Directories in dStorage
The ./zbox move
command can be used to move files between folder paths in dStorage. The parameters required for the move command can be viewed using ./zbox move --help
Parameter | Required | Description | default | Valid values |
---|---|---|---|---|
allocation | yes | allocation id | string | |
commit | no | save metadata to blockchain | false | boolean |
remotepath | yes | remote path of object to copy | string | |
destpath | yes | destination, an existing directory to copy object | string |
The required parameters are Allocation ID , dest path and remote path of the moving file . Steps:
Verify whether the directories are available in the remote path using the list command.
Response:
As you can see there are two directory paths available /myfiles
and myfiles1
.We will move the info.txt file available remotepath /myfiles
to the destpath
/myfiles1
.
2. Use the ./zbox move
command
Successful Response:
3. Verify whether the file has moved by listing the directory path using:
Response:
Renaming Files in dStorage
./zbox rename
command can be used to rename files already existing in dStorage.Only the allocation's owner can rename a file. Parameters for the command can be viewed using ./zbox rename --help
Parameter | Required | Description | default | Valid values |
---|---|---|---|---|
allocation | yes | allocation id | string | |
commit | no | save metadata to blockchain | false | boolean |
destname | yes | new neame of the object | string | |
remotepath | yes | remote path of object, do not use with authticket | string |
Command:
Response:
Verify by listing the remote path using
Response:
List files from Blobbers
./zbox list
command is used to list all the files from a specified directory on dStorage. Use ./zbox list --help
to know the possible parameters.
Parameter | Required | Description | Valid Values |
---|---|---|---|
allocation | yes | allocation id | string |
lookuphash | adjust storage expiration time | string | |
remotepath | yes | Remote Path of directory on dStorage | string |
Command :
Sample Response :
Get Stats for file in dStorage
./zbox stats
command can be used to get upload, download and challenge information for a file. Parameters for the command can be viewed using ./zbox stats --help
Parameter | Required | Description | default | Valid values |
---|---|---|---|---|
allocation | yes | allocation id | string | |
json | no | print result in json format | false | boolean |
remotepath | yes | file of which to get stats | string |
Steps:
Verify whether the file exists at the remote path using
Response:
2. Run the stats command using
Sample Response:
Get metadata for stored file
./zbox meta
command is used to get meta data for the given file in dStorage . Use ./zbox meta --help
to know more about possible flags.
Parameter | Required | Description | default | Valid values |
---|---|---|---|---|
allocation | yes | allocation id | string | |
authticket | no | auth ticked if not owner of the allocation | string | |
json | no | print result in json format | false | boolean |
lookuphash | no | hash of object, use with auth ticket | string | |
remotepath | no | remote path of object, do not use with authticket | string |
Command:
Sample Response:
Get Download cost
./zbox get-download-cost
determines the cost for downloading the remote file from dStorage. The client must either be the owner, a collaborator or be using an auth ticket.
Parameter | Required | Description | default | Valid values |
---|---|---|---|---|
allocation | yes | allocation id | string | |
authticket | no | auth ticket to use if not the owner | string | |
lookuphash | no | hash of remote file, use with auth ticket | string | |
remotepath | yes | file of which to get stats, use if owner | string |
Command:
Sample Response:
Get Upload cost
./zbox get-upload-cost
determines the cost for uploading a local file on dStorage. --duration
Ignored if --end
true, in which case the cost of upload calculated until the allocation expires.
Parameter | Required | Description | default | Valid values |
---|---|---|---|---|
allocation | yes | allocation id | string | |
duration | no | duration for which to upload file | duration | |
end | no | upload file until allocation expires | false | boolean |
localpath | yes | local of path to calculate upload | file path |
Command:
Response:
Get Differences
./zbox get-diff
command returns the differences between the local files specified by localpath
and the files stored on the root remotepath of the allocation.localcache
flag can also be specified to use the local cache of remote snapshot created during Sync for file comparison.
Parameter | Required | Description | default | Valid values |
---|---|---|---|---|
allocation | yes | allocation id | string | |
excludepath | no | remote folder paths to exclude during syncing | string array | |
localcache | no | local cache of remote snapshot | string | |
localpath | yes | local directory to sync | string |
Command:
Response :
Formatted Shortened Response :
Sync
./zbox sync
command is used to sync all files in the local folder. Creation, deletion and updation will be done automatically. Parameter details can be viewed using the ./zbox sync --help command.
Here are the Parameters:
Parameter | Required | Description | default | Valid values |
---|---|---|---|---|
allocation | yes | allocation id | string | |
commit | no | commet metadata to blockchain | false | boolean |
encryptpath | no | local directory path to be uploaded as encrypted | false | boolean |
excludepath | no | paths to exclude from sync | string array | |
localcache | no | local cache of remote snapshot. Used for comparsion with remote. After sync will be updated. | string | |
localpath | yes | local directory to which to sync | file path | |
uploadonly | no | only upload and update files | false | boolean |
Example:
Let's sync files stored in the local directory .The local directory for sync in the command below is mapped to local
variable. Variable cache
is marked to local cache text file located inside the local directory..
Command:
Response:
Verify whether all the files located in the local directory are synced to dStorage using
Batch upload files to dStorage
./zbox sync
command with --uploadonly
flag can be used to upload all the files in local folder to dStorage. Parameter details can be viewed using the ./zbox sync --help
Here are the Parameters:
Parameter | Required | Description | default | Valid values |
---|---|---|---|---|
allocation | yes | allocation id | string | |
commit | no | commet metadata to blockchain | false | boolean |
encryptpath | no | local directory path to be uploaded as encrypted | false | boolean |
excludepath | no | paths to exclude from sync | string array | |
localcache | no | local cache of remote snapshot. Used for comparsion with remote. After sync will be updated. | string | |
localpath | yes | local directory to which to sync | file path | |
uploadonly | no | only upload and update files | false | boolean |
Example:
We will create three txt files and move them into a directory to specify it as a local path for the ./zbox sync
command.
Here is the list of contents of the directory:
Now by using the ./zbox sync
command with --uploadonly
flag we will batch upload these files to the dStorage .Variable local
is marked to local directory containing all the files for upload. Variable cache
is marked to local cache text file located inside the local directory.
Command:
Response
Verify whether the files are uploaded using
Response:
All the files located in the local directory should be listed in the remotepath.
Private File Sharing
Private sharing of files can be implemented using the ./zbox share command
.Parameters can be viewed using./zbox share --help
are shown below:
Parameter | Required | Description | Valid values |
---|---|---|---|
allocation | yes | allocation id | string |
clientid | no | id of user to share file with, leave blank for public share | string |
encryptionpublickey | no | public key of the client to share file with, required if clientId | string |
expiration-seconds | no | seconds before | int |
remotepath | yes | remote path of file to share | string |
revoke | no | revoke share for remote path | flag |
Note: Private File Sharing only works with encrypted files.
Let's consider two users (User A and B) and User A wants to privately share the info.txt
file located in myfiles/info.txt
directory in dStorage with User B.
1.User A will upload the encrypted info.txt
file. For instructions on how to upload an encrypted file check Uploading an Encrypted file on dStorage.
2. User B has to share his client id and encryption public key which can be retrieved using thegetwallet
command. Here we are showing the output of the UserB getwallet
command
Command:
Response:
3.Let's export the User B clientid and public encryption key to a variable named UserB
and PUBLIC_KEY_UserB
4.Now use the ./zbox share
command with above exported variables .Also specify the allocation and remote path for the file shared by User A which is myfiles/info.txt
.
So the sample command for the UserA is:
Response:
The Response provides an encrypted auth token string that only UserB can use for private download. Any other user using this AuthToken will not be able to download the file.
5.Let's use ./zbox download
using Authtoken provided on UserB computer to privately download the file. Necessary Flags are Authtoken and local path to save the file. Make sure to point encrypted auth token using the export AUTH=<Auth token>
before running the download command.
Response:
6.To check sharing details and verify whether the file is encrypted. You can also base decode the Authtoken using:
Response:
The response will provide all the private sharing details and encryption key for the shared file under re_encryption_key
field.
Cancel Sharing of File for a Particular User
./zbox share --revoke
is used to cancel the share for particular buyer that performed by the seller through Private File Sharing.
Note : ./zbox share --revoke
only works for encrypted files on dStorage.
Required Parameters for Cancel Share are :
Parameter | Required | Description | Valid Values |
---|---|---|---|
allocation | yes | allocation id | string |
remotepath | yes | remote path of an encrypted file that was earlier shared with the user | duration |
clientid | yes | Wallet Client ID of the user whose share access has to be revoked | string |
Command :
Response :
Private Directory Sharing
Private sharing of directories can be implemented using the ./zbox share command
.Parameters can be viewed using./zbox share --help
are shown below:
Parameter | Required | Description | Valid values |
---|---|---|---|
allocation | yes | allocation id | string |
clientid | no | id of user to share file with, leave blank for public share | string |
encryptionpublickey | no | public key of the client to share file with, required if clientId | string |
expiration-seconds | no | seconds before | int |
remotepath | yes | remote path of file to share | string |
revoke | no | revoke share for remote path | flag |
Note: Private Directory Sharing works for both encrypted and non-encrypted files
Let's consider two users (User A and B) and User A wants to privately share the docs
directory in dStorage with User B.
1.User A should check whether the docs
directory exists on dStorage using the list command.
Command :
Response :
2. Now User B has to share his client id and encryption public key which can be retrieved using thegetwallet
command. Here we are showing the output of the UserB getwallet
command
Command:
Response:
3.Let's export the User B clientid and public encryption key to a variable named UserB
and PUBLIC_KEY_UserB
4.Now use the ./zbox share
command with above exported variables .Also specify the allocation and remote path for the directory .
Response:
The Response provides an encrypted auth token string that only UserB can use for private download. Any other user using this AuthToken will not be able to download the file.
5. The string is encoded , to know the sharing details of the string UserB can decode it using the command :
Response :
6. Now UserB can download files inside the shared directory with the AUTH token. First point the encrypted auth token to a variable using the export
command.
7. Download the files inside the directory by using the following command
Response:
Note : The allocation ID required in the command above can be retrieved from decoding the AUTH string. For remote path the actual path for the shared directory should be specified.
If you are not able to download files and getting Error in file operation: File content didn't match with uploaded file response check Error 2 in Tips and Troubleshooting Section.
Repair a file on dStorage
Use ./zbox start-repair
command to repair a file on dStorage. Parameters can be viewed using the --help
flag.
Parameter | Required | Description | default | Valid values |
---|---|---|---|---|
allocation | yes | allocation id | string | |
repairpath | yes | remote path of files on dStorage to repair | string | |
rootpath | yes | File path for local files. Files from rootpath will be compared with files on dStorage for data blocks error. | string |
Sample Command:
Response:
In the above response the total number of files repaired were 0 which means all the data blocks for the files are not altered or damaged.
Rollback
Use ./zbox rollback
to rollback to a previous state of allocation. This is helpful when you want to rollback to previous version of files you updated on allocation using Update allocation.
Parameter | Description | Valid Values |
---|---|---|
--allocation | Provide Allocation ID for rollback. | string |
--h,--help | help for rollback | int |
Sample Command:
Sample Response:
Total Stored Data
./zbox total-stored-data
returns the summation of all the Size fields in all the files stored across blobbers.
Sample Command:
Sample Response:
Asking the receiver to pay for download (RX_pay)
RX pay is a specified boolean flag in ./zbox download
command which asks the file receiver to pay for their file download.
Parameters for --rx_pay
flag can be viewed using the zbox download --help
.
Parameter | Required | Description | Default | Valid values |
---|---|---|---|---|
allocation | yes | allocation id | string | |
authticket | no | auth ticked if not owner of the allocation, use share to get auth ticket | string | |
blockspermarker | no | download multiple blocks per marker | 10 | int |
commit | no | save metadata to blockchain | false | boolean |
endblock | no | download until specified block number | int | |
localpath | yes | local path to which to download the file to | file path | |
remotepath | yes | remote path to which the file was uploaded | string | |
rx_pay | no |
| false | boolean |
startblock | no | start download from specified block | int | |
thumbail | no | only download the thumbnail | false | boolean |
As we can see that RX pay requires download by auth ticket and a rx_pay
boolean set to true .
Example: Let's take two users (User A and User B). User A is the seller of the file downloaded by User B (receiver).
1.User A will upload a file on dstorage and share an authtoken for the file. For example we are uploading
info1.txt
file located on myfiles
directory of dStorage which User B will download and pay for himself using --rxpay.
2.To make the file available for pay User A has to set the--attr-who-pays-for-reads
for upload command to 3rd_party (by default it is set to "owner")
Sample Command:
3.To verify whether downloads payer are properly configured use:
Response:
In the response, you will notice that the downloads payers section for info1.txt
are successfully set to 3rd-party.
4.To know how to share a file on dStorage and get an AuthToken for a file have a look at the sharing a file on dStorage .
5.Once the auth token is created and marked to a variable, share it with the receiver(UserB) who will run the .zbox download
command to download the file with --rx_pay
flag.
Command for User B:
Response:
To verify whether (UserB) has paid successfully for the download check the wallet and read pool balance for change using ./zwallet getbalance
and /zbox rp-info
.
Note: Do unlock read tokens using rp-unlock
before using ./zwallet getbalance
to let the download money reflect in wallet.
Video Streaming
Video streaming with Zboxcli can be implemented with players for different operating platforms(iOS, Android, Mac). Zbox CLI does not have a player itself and uses the following helping functions :
downloadFileByBlocks - properly returns file-chunks with correct byte range,.
getFileMeta - returns actuaBlockNumbers and actualFileSize (exclude thumbnail size)
getFileMetaByAuth - same updates as getFileMeta
listAllocation - returns actuaBlockNumbers and actualFileSize (exclude thumbnail size)
For all the Platforms streaming implementation using ZboxCLi has the following flow:
When the user starts the video player (ExoPlayer for Android or AVPlayer for iOS), A ZChainDataSource starts chunked download and requests chunks of video from the buffer(a Middleman between streaming player and Zbox).
After the arrival of the first chunk, the player starts requesting more chunks from the buffer which request the Zbox SDK .Zbox SDK which is built using GO makes use of the downloadFileByBlocks method to reliably download large files by chunking them into a sequence of parts that can be downloaded individually. Once the blocks are downloaded they are read into input streams and added to the media source of the streaming player.
The task of downloading files and writing them to buffer using ZboxSDK happens constantly and If player request random bits of video they are delivered instantly by the buffer.
In a case, if the player didn't receive chunks (for example it's still not downloaded), then the player switches to STALE state and the video stream will pause. During the STALE state, player tries to make multiple requests for chunks, if didn't receive a response the video stream stops.
To understand how Zbox CLI provides downloading of files by blocks check the below demonstration:
For instance, Let's consider an allocation which has a audio.mp3
file stored on dStorage. Make sure the file has a large size(more than 64 kB(64000 bytes) ) to download the file by blocks. The size and other attributes of the sampleaudio.mp3
file can be viewed using
Response:
Here we can see audio.mp3
file has size (5993286) bytes and has 92 blocks. If we want to download a certain number of blocks for the info.txt file we can use --endblock
or --startblock
flag with ./zbox download
command. Other flags for download can be viewed using ./zbox download --help
Parameter | Required | Description |
---|---|---|
--allocation | yes | Allocation ID |
--authticket | no | Auth ticket for the file to download if you dont own it |
-b, --blockspermarker | no | Pass this option to download multiple blocks per marker (by default 10) |
--delay | no | Pass segment duration to generate media playlist(m3u8). only works with --live. default duration is 5s. (default 5) |
-e,--endblock | yes | Pass this option to download till specific block number |
-h, --help | no | help for download |
--live | no | Start m3u8 downloader,and automatically generate media playlist(m3u8) on --localpath |
--localpath | yes | Local path of file to download |
--lookuphash | no | The remote lookuphash of the object retrieved from the list |
--remotepath | yes | Remote path to download |
-s, --startblock | yes | Pass this option to download from specific block number |
-t,--thumbnail | no | Pass this option to download only the thumbnail |
To download only 3 blocks of audio.mp3
file we specify --startblock
and--endblock
with an integer value of 1 and 3. --blockspermarker
flag can also be specified to download multiple blocks at a time(default is 10).
The whole download command for downloading till 3rd block of audio.mp3
file would be:
Response:
As we can see, the downloaded file size(393216) is quite less than the original(2996198) which means zbox has downloaded some blocks of the file.
Feed
Use feed
command to automatically download segment files from remote live feed with --downloader-args "-q -f best"
encode them into new segment files with
--delay
and--ffmpeg-args
, and upload.please use
youtube-dl -F https://www.youtube.com/watch?v=pC5mGB5enkw
to list formats of video (see below).
--downloader-args "-f 22"
dowloads video with 22 mp4 1280x676 720p 2117k , avc1.64001F, 30fps, mp4a.40.2 (44100Hz) (best)
The user must be the owner of the allocation.You can request the file be encrypted before upload, and can send thumbnails with the file.
Parameter | Required | Description | Default | Valid values |
---|---|---|---|---|
allocation | yes | allocation id, sender must be allocation owner | string | |
encrypt | no | encrypt file before upload | false | boolean |
localpath | yes | local path of segment files to download, generate and upload | file path | |
remotepath | yes | remote path to upload file to, use to access file later | string | |
thumbnailpath | no | local path of thumbnaSil | file path | |
chunknumber | no | how many chunks should be uploaded in a http request | 1 | int |
delay | no | set segment duration to seconds. | 5 | int |
feed | no | set remote live feed to url. | false | url |
downloader-args | no | pass args to youtube-dl to download video. default is "-q -f best". | -q -f best | |
ffmpeg-args | no | pass args to ffmpeg to build segments. | -loglevel warning |
Sample Command:
Last updated