Reflections from the Frontier

Jennifer Mack, mack[at], and Norman Grogin, nagrogin[at]

The Frontier Fields (FF) program was designed to push Hubble’s capabilities to the extreme, by using the power of gravitational lensing to see further back in time than ever before. By equal measure, the project has pushed the limits of our understanding of Hubble’s two main imaging cameras and has led to new initiatives that ultimately benefit science for the entire user community. After completing its final observation in September 2016, logging nearly 900 orbits over three years, the team celebrated with a champagne toast to reflect on the success of the program and recalled the many lessons learned and the evolution of teamwork that made the program so productive.

The FF project has involved tremendous coordination between many separate teams, including not only a Hubble component, but also complimentary observations from Spitzer and Chandra at infrared and X-ray wavelengths. A prior newsletter article (Lotz et al. 2013) describes the early planning stages of the project, including the target selection and careful design of the observing program. Lensing models were solicited from several independent teams prior to commencing observations in order to facilitate rapid analysis by the scientific community. The FF team concentrated for many months to develop a rigorous data pipeline, using the full set of Hubble Ultra-Deep Field (UDF) observations to optimize strategies for alignment and image combination with the latest DrizzlePac software, which had been released only a year prior. The goal was to deliver high-level, robust data products in a short turnaround time, while keeping the scientific community engaged and apprised of the latest results. This strategy follows a tradition set by prior deep-field programs like the Hubble Deep Field and the UDF.

Figure 1: The Frontier Fields Implementation Team.

As the data started pouring in, a well-choreographed performance began to unfold at the Institute. Once the observing parameters for each new target were submitted, instrument contact scientists carefully reviewed the implementation strategy and collectively provided suggestions to further improve data quality. Schedulers then worked together with those developing the telescope's long-range plan in order to fit each set into a 6–8 week observing window, or epoch, limited by the requirement to keep the telescope at a fixed orientation for each target. As the observations began to execute, the archive team "fast-tracked" the data onto dedicated FF machines for inspection by instrument experts. Due to the limited visibility, timeliness was of the essence, so any data-quality issues were quickly reported, ensuring that repeated observations could be acquired before the end of the window. The data pipeline team worked closely with both the Advanced Camera for Surveys (ACS) and Wide Field Camera 3 (WFC3) instrument teams to provide feedback on any limitations of the existing calibration and software tools. Only a few weeks following execution, the team delivered the first versions of high-level data products to the archive and announced their availability to the community. To flawlessly execute an observing program spanning three years over twelve separate epochs, team stamina is imperative, and everyone worked tirelessly to deliver the best quality data products possible.

In the first six months of observations, a large number of infrared (IR) exposures were noted to have strong time-varying background caused by line emission in the Earth’s atmosphere (Brammer et al. 2014). The FF team worked with WFC3 instrument scientists and schedulers to calculate each target's ephemeris and predict which portion of Hubble's orbit would be affected. The exposures were then tactically arranged within the orbit to shield observations in the vulnerable F105W filter. The implementation of these scheduling improvements in mid-2014 considerably reduced the excess background measured in the following two years of observations. The FF and WFC3 teams worked together to develop new strategies for removing time-varying IR background and to share those recommendations with Hubble users who could benefit from manual reprocessing their data (Robberto 2014; Brammer 2016).

Also noted in the early stages of the project were a handful of WFC3 visits severely impacted by IR persistence, a residual decaying signal from observations obtained just prior to ours. The worst of these were scanned grism observations in the MACS0416 parallel field, which affected a large rectangular region at the center of the detector. The FF team worked closely with WFC3 experts, the Hubble Mission Office, and the long-range planning team to develop new strategies to mitigate persistence. As a result, in mid-2014 the WFC3 team began flagging all IR programs predicted to cause significant persistence, with the goal of protecting similar susceptible science programs classified as "sensitive actors." Any programs expected to saturate >1% of the detector (e.g., deep exposures of star clusters) were categorized as "bad actors" and given a two-orbit scheduling buffer. Programs flagged as "worst actors"—typically, spatial scans and IR persistence calibration programs—were given a 10-orbit buffer. This new strategy significantly reduced persistence in the remaining FF epochs and had a similar benefit for protecting other sensitive programs.

The teams combined forces to measure the IR persistence decay rate and to understand why the residual signal seemed to rise again after obtaining a high-signal image such as an internal lamp flat. Investigations probed how well various persistence models (Long et al. 2013, 2015) predicted the residual signal measured in FF data for different types of prior observations. For example, the level of persistence predicted for scanned grism programs was often much lower than what was measured directly from the FF data. For this reason, the persistence-corrected data products were not directly used in the pipeline, but to define masks for excluding contaminated pixels prior to stacking the data instead.

In addition to the standard calibration pipeline, CALACS, new standalone software was developed for improving the FF ACS mosaics. Included is a new "self-calibration" procedure to refine the dark-current subtraction for programs with many dithered frames at the same orientation. This software does a better job of identifying and removing warm pixels and flagging the charge-transfer efficiency (CTE) trails behind bright cosmic rays. As a result, self-calibration removes coherent residual structure, reduces the total noise, and increases the overall depth of the combined ACS mosaics (Ogaz et al. 2015). Early FF results showed that self-calibration was most effective when combining as many frames as possible within an anneal (heating) cycle. Starting in the spring of 2014, the team worked with schedulers to strategically arrange ACS/WFC imaging such that observations using the same filter were grouped within the same anneal period. After rigorous testing and a user "cookbook" developed by the FF team, the software was provided for download from the ACS selfcal webpage for use with any ACS/WFC or WFC3/UVIS imaging programs.

In another example, the FF team worked closely with the ACS team to test improvements to the dark calibration. In January 2015, the ACS team began adding LED post-flash to its calibration darks to preserve warm pixels previously lost to poor CTE (Ogaz et al. 2015). The FF team noted soon thereafter that the combined drizzled data products had significantly higher noise when using the new post-flashed darks for calibration. To remedy this, the ACS and FF teams together developed a new strategy for making calibration darks. By doubling the time-averaged baseline to compensate for additional noise in the flashed darks, the new ACS "super darks" have much lower noise than the original two-week baseline. After verifying the improvement using FF data in early 2016, the ACS team retroactively applied this new approach to all flashed darks since 2015, with an improved set of reference files to replace the prior set.

Pushing the depth of the target fields even further, the pipeline team combined all archival ACS and WFC3 observations with those from the dedicated FF programs. Because the ACS geometric distortion has a strong linear component that changes with time (Anderson 2007; Ubeda et al. 2013), an accurate correction model is fundamental for combining observations over a long time-baseline. Members of the FF team worked closely with ACS instrument experts to test improvements to the distortion model (Borncamp et al. 2015). After considerable work by both groups, the ACS team delivered an improved set of reference files in mid-2015 (Kozhurina-Platais et al. 2015). These new solutions remove systematic skew terms apparent in the FF astrometric residuals when using prior solutions.

The FF observations were designed to exquisitely subsample the point spread function, using a precise sub-pixel dither for each visit, with subsequent visits slightly offset to maximally sample the WFC3/IR pixel phase. These data were used to demonstrate to Hubble users how to optimize the pixel sampling of the observatory's detectors when combining dithered observations with AstroDrizzle (Avila et al. 2015). By inspecting differences between the "visit-level" drizzled products and the combined massively dithered drizzled mosaics, the team was able to quickly spot any issues with the image alignment, residual unmasked IR persistence, or frequent asteroids seen whizzing through the images. As a result of this work, new software to automatically detect and flag satellites in Hubble observations (Borncamp & Lim 2016) was developed and provided to the community, making it far less cumbersome for Hubble users to remove these types of artifacts from their data.

Figure 2: Combined mosaic of Abell S1063 from the FF program (24 orbits) compared to CLASH (two orbits), showing the dramatic improvement in depth and image quality for the reddest F160W filter.

In addition to all the intense technical work associated with FF, the Office of Public Outreach played a vital role in the project from the beginning, with a public blog focusing on the science gains possible when using gravitational lensing to further extend Hubble's range and engaging the community with informative videos and online hangouts. A complementary science blog, titled "A Sneak Peek at the First Billion Years of the Universe" provided highlights on the latest calibration methods, announcements of new data products, and the latest science results from astronomers across the globe.

In the end, the investment of orbits and personnel toward the FF project has had immense benefits for general Hubble science. These include improvements in scheduling, new calibrations and software, and a deeper understanding of the instruments. Ambitious projects like this will continue to push our understanding of the observatory’s instruments and drive the need for new methodologies that will benefit Hubble science for years to come. Future projects like WFIRST are looking at FF as a model for producing high-level data products driven by astronomy community needs and Institute know-how.


Anderson, J. 2007, ACS ISR 2007-08, "Variation of the Distortion Solution of the WFC"

Avila, R. J., Koekemoer, A., Mack, J., & Fruchter, A. 2015, ACS ISR 2015-01, "Optimizing pixfrac in Astrodrizzle: An example from the Hubble Frontier Fields"

Borncamp, D., Kozhurina-Platais, V., & Avila, R. 2015, ACS ISR 2015-02, "Results of the Updated ACS/WFC Distortion Correction"

Borncamp, D., & Lim, P. 2016, ACS ISR 2016-01, "Satellite Detection in Advanced Camera for Surveys/Wide Field Channel Images"

Brammer, G. 2016, WFC3 ISR 2016-16, "Reprocessing WFC3/IR Exposures Affected by Time-Variable Backgrounds"

Brammer, G., Pirzkal, N., McCullough, P., & MacKenty, J. 2014, WFC3 ISR 2014-03, "Time-varying Excess Earth-glow Backgrounds in the WFC3/IR Channel"

Kozhurina-Platais, V., Borncamp, D., Anderson, J., Grogin, N., & Hack, W. 2015, ACS ISR 2015-06, "ACS/WFC Revised Geometric Distortion for DrizzlePac"

Long, K. S., Baggett, S. M., & MacKenty, J. W. WFC3 ISR 2013-07, "Characterizing Persistence in the WFC3 IR Channel: Observations of Omega Cen"

Long, K. S., Baggett, S. M., & MacKenty, J. W. WFC3 ISR 2015-15, "Persistence in the WFC3 IR Detector: An Improved Model Incorporating the Effects of Exposure Time"

Lotz, J., Reid, N., & Sembach, K. 2013, "Hubble Boldly Goes"

Ogaz, S., Avila, R., & Hilbert, B. 2015, "ACS/WFC3 Cal Improvements: Lessons Learned from the FF"

Ogaz, S., Anderson, J., & Golimowski, D. 2015, ACS ISR 2015-03, "Post-Flash Calibration Darks for the Advanced Camera for Surveys Wide Field Channel (ACS/WFC)"

Robberto, M. 2014, Proceedings of the SPIE, Volume 9143, id. 91433Z, "A generalized least square algorithm to process infrared data taken in non-destructive readout mode"

Ryan, R. E., & Baggett, S. M. 2015, WFC3 ISR 2015-11, "The Internal Flat Fields for WFC3/IR"

Ubeda, L., & Kozhurina-Platais, V., 2013, ACS ISR 2013-03, "ACS/WFC Geometric Distortion: A Time Dependency Study"