Skip to content
This repository was archived by the owner on Nov 28, 2025. It is now read-only.

Support compression for output in MR v1.#147

Open
shishaochen wants to merge 1 commit intotensorflow:masterfrom
shishaochen:output-codec
Open

Support compression for output in MR v1.#147
shishaochen wants to merge 1 commit intotensorflow:masterfrom
shishaochen:output-codec

Conversation

@shishaochen
Copy link

@shishaochen shishaochen commented Nov 5, 2019

Aligning with org.tensorflow.hadoop.io.TFRecordFileOutputFormat, enable compression in org.tensorflow.hadoop.io.TFRecordFileOutputFormatV1 as well.
To activate compression in old MapReduce APIs, simply specify options as below:

-Dmapred.output.compress=true
-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec

This pull request can be considered as supplementary to #61.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants